StarCoder2 and The Stack v2 Release

BigCode has released StarCoder2, a family of open large language models (LLMs) for code, alongside The Stack v2, a significantly expanded dataset for code pretraining. This release provides the community with transparently trained models and the underlying data and processing code to advance open-source code intelligence.

StarCoder2 Model Family and Specifications

StarCoder2 is available in three parameter sizes, each trained on different subsets of The Stack v2 and supported by different partners:

• StarCoder2-3B: Trained by ServiceNow on 3+ trillion tokens across 17 programming languages.
• StarCoder2-7B: Trained by Hugging Face on 3.5+ trillion tokens across 17 programming languages.
• StarCoder2-15B: Trained by NVIDIA using NVIDIA NeMo on NVIDIA accelerated infrastructure. This flagship model was trained on 4+ trillion tokens and covers 600+ programming languages.

All models in the family share a common architecture featuring Grouped Query Attention (GQA), a context window of 16,384 tokens, and a sliding window attention of 4,096 tokens. They were all trained using the Fill-in-the-Middle (FIM) objective to improve code completion capabilities.

Performance Benchmarks

StarCoder2-15B is positioned as the best model in its size class, matching the performance of models with 33B+ parameters on many evaluations. Additionally, the StarCoder2-3B model matches the performance of the previous generation StarCoder1-15B.

The Stack v2 Dataset

The Stack v2 is the largest open code dataset designed for LLM pretraining, derived from the Software Heritage archive—a non-profit initiative by Inria and UNESCO to preserve all publicly available software source code.

Compared to its predecessor, The Stack v2 offers substantial increases in scale and quality:

Metric	The Stack v1	The Stack v2
Full Size	6.4TB	67.5TB
Deduplicated Size	2.9TB	32.1TB
Training Dataset	~200B tokens	~900B tokens

Technical improvements in v2 include an enhanced language and license detection procedure, better filtering heuristics, and a structure that groups data by repositories. The latter allows models to be trained with repository-level context, improving their understanding of project-wide dependencies.

Governance and Accessibility

StarCoder2 and its associated datasets are released under the BigCode OpenRAIL-M v1 license agreement. To ensure transparency and data governance, BigCode provides several tools:

• StarCoder2 Search: A full-text search tool for code within the pretraining dataset.
• StarCoder2 Membership Test: A high-speed utility to check if specific code was present in the pretraining data.

BigCode is a scientific collaboration led by Hugging Face and ServiceNow dedicated to the responsible development of open code LLMs.