StarCoder Release Notes
Hugging Face and ServiceNow's BigCode collaboration has released StarCoder and StarCoderBase, two Large Language Models (LLMs) for code. These 15B parameter models are trained on 1 trillion tokens of permissively licensed data, providing a high-performance, open alternative to closed models like OpenAI's code-cushman-001.
Model Architecture and Training
StarCoderBase is the foundational model trained on a dataset comprising 80+ programming languages, Git commits, GitHub issues, and Jupyter notebooks. StarCoder is a specialized version of StarCoderBase that has been further fine-tuned on 35B Python tokens.
Key technical specifications include:
- Parameter Count: Approximately 15 billion.
- Training Volume: 1 trillion tokens.
- Context Length: Over 8,000 tokens, which exceeds the input capacity of other open Code LLMs at the time of release.
Performance Benchmarks
StarCoder and StarCoderBase demonstrate superior performance compared to various open-access and closed models across multiple benchmarks.
Python Proficiency (HumanEval and MBPP)
On the HumanEval benchmark, which tests function completion based on signatures and docstrings, StarCoder and StarCoderBase outperform models such as PaLM, LaMDA, and LLaMA. When using a specific prompt to bypass the model's tendency to produce placeholder comments (e.g., # Solution here), the "StarCoder-Prompted" version achieved a state-of-the-art result for open models with a score of 40.8% on HumanEval.
| Model | HumanEval | MBPP |
|---|---|---|
| LLaMA-65B | 23.7 | 37.7 |
| PaLM-540B | 26.2 | 36.8 |
| StarCoderBase | 30.4 | 49.0 |
| code-cushman-001 | 33.5 | 45.9 |
| StarCoder | 33.6 | 52.7 |
| StarCoder-Prompted | 40.8 | 49.5 |
Multilingual and Data Science Capabilities
StarCoder matches or outperforms code-cushman-001 across many languages on the MultiPL-E benchmark. Additionally, StarCoder outperforms all other open-access models on the DS-1000 data science benchmark.
Technical Capabilities and Applications
Beyond simple code completion, the StarCoder models support several advanced developer workflows:
- Technical Assistance: By utilizing the Tech Assistant Prompt, the models can answer programming-related requests and act as a technical assistant.
- Code Modification: The models can modify existing code based on natural language instructions.
- Code Explanation: The models can explain code snippets in natural language.
- Autocomplete: The models provide standard code autocompletion functionality.
Data Governance and Safety
StarCoder was trained on a subset of "The Stack 1.2," consisting exclusively of permissively licensed code. The BigCode project implemented several safety and governance measures:
- PII Redaction: In collaboration with Toloka, Personal Identifiable Information (PII) such as names, passwords, and email addresses were removed from the training data.
- Opt-out Process: Code contributors can remove their data from the dataset via a dedicated opt-out process.
- Attribution Tool: A code attribution tool is provided to find generated code within the training dataset.
- Licensing: The models are released under an improved version of the OpenRAIL license to simplify integration for corporate products.
Ecosystem and Available Resources
BigCode has released a comprehensive suite of tools alongside the model weights:
- Models: StarCoder (Python-tuned), StarCoderBase (Multilingual), StarEncoder (Encoder model), and StarPii (PII detector).
- Tools: A VSCode extension, StarCoder Chat, a playground, and a dedicated code editor.
- Datasets: The full preprocessed training dataset and a new PII dataset for training and evaluating PII removal.