Cosmopedia: Large-Scale Synthetic Data for LLM Pre-training
Hugging Face has released Cosmopedia, a large-scale synthetic dataset designed to replicate the training methodology of Microsoft's Phi models. The dataset contains over 30 million files and 25 billion tokens, making it the largest open synthetic dataset available for pre-training Large Language Models (LLMs) from scratch.
Scaling Synthetic Data for Pre-training
While synthetic data is commonly used for instruction-tuning, Cosmopedia focuses on the challenge of scaling from a few thousand samples to millions of high-quality samples for pre-training. This effort was motivated by the performance of the Phi series, which demonstrated that models trained predominantly on synthetic "textbook-style" data can surpass larger models trained on general web datasets.
Methodology for Dataset Generation
Cosmopedia was generated using Mixtral-8x7B-Instruct-v0.1. The core challenge was maintaining diversity and minimizing duplicate content across 30 million prompts. Hugging Face employed three primary strategies for prompt curation:
1. Curated Sources
Educational resources from Stanford courses, Khan Academy, OpenStax, and WikiHow were used to extract outlines and topics. To maximize the utility of these limited sources, the team leveraged diversity in audience and style. A single topic was repurposed for different target audiences (young children, high school students, college students, and researchers) and different formats (textbooks, blog posts, and wikiHow articles).
2. Web Data
Web data contributed over 80% of the prompts. The team clustered millions of web samples from datasets like RefinedWeb into 145 clusters. Mixtral was then used to identify the common topic of each cluster. Low-educational-value content, such as celebrity gossip and adult material, was filtered out, leaving 112 retained topics. Prompts were then built by instructing the model to generate textbooks based on web samples within these topics.
3. Instruction Datasets and Stories
To address a lack of common sense and fundamental grade-school knowledge in initial models, the team used UltraChat and OpenHermes2.5 as seed data to generate stories incorporating day-to-day knowledge.
Technical Stack and Implementation
Generating 25 billion tokens required over 10,000 GPU hours on H100 GPUs. The technical implementation included:
- Text Clustering: The
text-clusteringrepository was used to organize web data. - Generation at Scale: The
llm-swarmlibrary was used to deploy Mixtral-8x7B locally using TGI (Text Generation Inference) and vLLM. - Decontamination: To prevent benchmark leakage, a pipeline using 10-gram overlap and
difflib.SequenceMatcherwas implemented. Samples were discarded if the match ratio to benchmark samples exceeded 0.5. - Training and Evaluation: A 1B parameter model (cosmo-1b) was trained using the Llama2 architecture via the
nanotronlibrary and evaluated usinglighteval.
Performance of Cosmo-1B
Evaluation of the cosmo-1b model showed that it performs better than TinyLlama 1.1B on ARC-easy, ARC-challenge, OpenBookQA, and MMLU. It is comparable to Qwen-1.5-1B on ARC-challenge and OpenBookQA, though it still shows performance gaps compared to Phi-1.5.
Future Directions
Hugging Face identifies hallucinations in historical facts and mathematical reasoning as a primary area for improvement. Proposed solutions include using Retrieval Augmented Generation (RAG) to incorporate external information from sources like Wikipedia during the generation process and implementing hallucination measurement methods to identify problematic domains.