Code Llama Release Notes

Code Llama is a family of state-of-the-art, open-access large language models specialized for coding tasks, built upon the Llama 2 architecture. These models are designed to enhance developer productivity by automating code completion, generating unit tests, and writing documentation.

Model Variants and Training

Code Llama is available in three parameter sizes: 7 billion, 13 billion, and 34 billion. The models were developed through a multi-stage training process:

  • Base Models: Initialized from Llama 2 and trained on 500 billion tokens of near-deduplicated, publicly available code and related natural language discussions.
  • Python Specialist: A version of the base model further trained on an additional 100 billion tokens of Python code.
  • Instruction Fine-tuned: A version optimized to understand and follow natural language instructions. This variant was trained using a combination of the Llama 2 Chat instruction tuning dataset and a self-instruct dataset created by Llama 2 (generating interview questions) and Code Llama (generating solutions and unit tests).

Technical Capabilities

Long Context Window

Code Llama supports a native context window of 16,000 tokens. Through long-context fine-tuning and frequency domain RoPE scaling, the models can extrapolate to manage a context window of up to 100,000 tokens.

Code Infilling

The 7B and 13B base and instruction variants support code infilling. This allows the model to generate code that fits between an existing prefix and suffix, which is critical for cursor-based code assistants in IDEs. This capability is not available in the 34B models or the Python-specialized versions.

Language Support

The models demonstrate state-of-the-art performance across several major programming languages, including Python, C++, Java, PHP, C#, TypeScript, and Bash.

Deployment and Integration

Hugging Face Ecosystem

Code Llama is integrated into the transformers library (version 4.33+), providing support for:

  • 4-bit Loading: Integration with bitsandbytes allows the 34B model to run on consumer GPUs, such as the NVIDIA RTX 3090.
  • Precision Options: While trained in bfloat16 (recommended for fine-tuning), the models can be run in float16 for faster inference without discernible degradation in evaluation metrics.

Production Inference

For production-ready deployment, the models are compatible with Text Generation Inference (TGI) and Hugging Face Inference Endpoints. Recommended hardware for Inference Endpoints includes:

  • 7B models: 1x Nvidia A10G (GPU medium).
  • 13B models: 1x Nvidia A100 (GPU xlarge).
  • 34B models: 1x Nvidia A100 with bitsandbytes quantization (GPU 1xlarge) or 2x Nvidia A100 (GPU 2xlarge).

Performance Evaluation

Code Llama models are benchmarked using HumanEval and MultiPL-E (which translates HumanEval to multiple languages). According to the multilingual code leaderboard at the time of publication, the Code Llama family shows competitive performance across different sizes and flavors. For example, the CodeLlama-34B-Instruct model achieved a Python score of 50.79 and a JavaScript score of 45.85, with a leaderboard average score of 35.09.

Sources