StarCoder2-15B-Instruct-v0.1 release notes / what's new

Hugging Face has released StarCoder2-15B-Instruct-v0.1, the first code large language model (LLM) to be entirely self-aligned using a fully transparent and permissive pipeline. This model demonstrates that high-performance code generation can be achieved without relying on human-annotated data or distillation from proprietary teacher models like GPT-4.

Performance Benchmarks

StarCoder2-15B-Instruct-v0.1 achieves a 72.6 HumanEval score, surpassing CodeLlama-70B-Instruct's score of 72.0. According to evaluations on the EvalPlus benchmark, it is the top-performing permissive LLM at its scale, outperforming Grok-1, Command-R+, DBRX, and closely matching Snowflake Arctic 480B and Mixtral-8x22B-Instruct.

Further evaluations on LiveCodeBench—which uses coding problems created after September 1, 2023—show that StarCoder2-15B-Instruct-v0.1 consistently outperforms OpenCodeInterpreter-SC2-15B, a model trained on data distilled from GPT-4. This suggests that LLMs may learn more effectively from data within their own distribution than from a shifted distribution provided by a teacher model.

The Self-Alignment Pipeline

The training pipeline for StarCoder2-15B-Instruct-v0.1 consists of three primary stages: seed collection, instruction generation, and response validation.

1. Collecting Seed Code Snippets

To ensure a diverse set of programming principles, the team extracted Python functions with docstrings from The Stack v1. The resulting dataset of 250k Python functions was derived from an initial 5M functions using the following filters:

  • Type Checking: Pyright heuristic type-checker was used to remove functions with static errors.
  • Decontamination: Exact string matching was used to remove benchmark items from the training set.
  • Docstring Quality: StarCoder2-15B acted as a judge to filter out poorly documented functions.
  • Near-Deduplication: MinHash and locality-sensitive hashing (Jaccard similarity threshold of 0.5) were used to remove duplicates.

2. Self-OSS-Instruct

Using in-context learning with 16 few-shot examples, the base StarCoder2-15B model self-generated instructions through a two-step process:

  • Concepts Extraction: The model identified foundational programming principles (e.g., pattern matching, data type conversion) within the seed functions.
  • Instruction Generation: The model generated a coding task based on those identified concepts.

This process resulted in the generation of 238k instructions.

3. Response Self-Validation

Rather than distilling responses from a teacher model, StarCoder2-15B was instructed to generate both a response and corresponding tests for self-validation.

For each instruction, the model generated 10 samples. These samples were executed in a sandbox environment, and only those that passed the execution tests were retained. Out of 2.4M generated responses, 500k passed the execution test. After deduplication, a final SFT (Supervised Fine-Tuning) dataset of 50k instructions, each paired with a random passing response, was created.

Transparency and Open Source Resources

Unlike many permissive models that lack transparency regarding their training data and pipelines, StarCoder2-15B-Instruct-v0.1 is fully transparent. Hugging Face has open-sourced the following resources:

Sources