bespokelabsai/curator
Synthetic data curation for post-training and structured data extraction
What it solves
Bespoke Curator is a Python library designed to simplify the creation of synthetic data pipelines. It addresses the difficulty of generating, curating, and scaling high-quality datasets for post-training tasks such as model fine-tuning, distillation, and structured data extraction.
How it works
Curator provides a high-level curator.LLM class that handles the complexities of bulk inference, including parallelism, retries, and caching. Users define a prompt method for input processing and a parse method to handle structured outputs (via Pydantic). It supports a wide range of of inference backends including OpenAI, Anthropic, Gemini, vLLM, and Ollama. To reduce costs, it includes built-in support for various provider batch APIs.
Who it’s for
It is intended for AI researchers and developers who need to generate large-scale synthetic datasets for training reasoning models, fine-tuning LLMs (e.g., via LoRA), or extracting structured information from unstructured text.
Highlights
- Scalable Bulk Inference: Built-in performance optimizations for asynchronous operations and fault recovery.
- Structured Outputs: First-class support for Pydantic-based structured data generation.
- Batch Mode: Integration with OpenAI, Anthropic, Gemini, and others to cut token costs by up to 50%.
- Fine-Tuning Integration: Direct pipelines to LoRA fine-tuning via the Tinker SDK and managed SFT on Fireworks AI.
- Code Execution: Ability to run generated code using backends like Ray, Docker, or e2b.
- Data Monitoring: Includes a viewer to monitor data generation in real-time.