CodeGemma Release Notes / What's New

Google has released CodeGemma, a family of open-access large language models (LLMs) specialized for code. Built upon the pre-trained 2B and 7B Gemma checkpoints, CodeGemma models are further trained on 500 billion additional tokens of English language data, mathematics, and code to enhance logical and mathematical reasoning capabilities for code completion and generation.

CodeGemma Model Variants

CodeGemma is available in three distinct flavors to suit different deployment and use-case requirements:

  • CodeGemma 2B: A base model trained exclusively on Code Infilling. It is optimized for fast code completion and generation, particularly in environments where low latency or high privacy are prioritized.
  • CodeGemma 7B: A base model trained on a mix of code infilling data (80%) and natural language. It supports code completion, as well as general code and language understanding and generation.
  • CodeGemma 7B Instruct: A version of the 7B model fine-tuned for instruction following. It is designed for conversational interactions regarding programming, code, and mathematical reasoning.

All models in the family maintain an 8K token context size.

Performance and Benchmarks

CodeGemma-7B demonstrates strong performance against other 7B-class models. On the HumanEval benchmark for Python, it outperforms similarly sized models with the exception of DeepSeek-Coder-7B. This performance trend extends to other programming languages including Java, JavaScript, and C++ via the MultiPL-E translation of HumanEval.

According to the technical report, CodeGemma-7B performs best among 7B models on the GSM8K benchmark. Additionally, the CodeGemma-7B-it (instruct) version shows improvements in the most popular programming languages on both HumanEval and MBPP.

Technical Implementation and Prompting

Code Infilling (FIM)

CodeGemma 2B and 7B utilize a Fill-In-the-Middle (FIM) objective for code completion. This allows the model to generate code based on a prefix and a suffix to fill the gap between them. The following special tokens are used to structure these prompts:

  • <|fim_prefix|>: Precedes the context before the cursor.
  • <|fim_suffix|>: Precedes the suffix; this token marks the exact position where the model should generate code.
  • <|fim_middle|>: The prompt that triggers the generation.
  • <|file_separator|>: Used to provide multi-file contexts.

Instruction Tuning

CodeGemma 7B Instruct follows the standard Gemma instruction-tuned prompt format, using a conversation structure with <bos>, <start_of_turn>user, and <start_of_turn>model tags.

Deployment and Integration

Hugging Face Transformers

CodeGemma is integrated with the transformers library (release 4.39+). It supports safetensors format, 4-bit quantization via bitsandbytes, Parameter-Efficient Fine-Tuning (PEFT), and Flash Attention 2. The models are compatible with torch.compile() for inference speedups.

Hardware Requirements:

  • CodeGemma 2B: Requires approximately 6 GB of RAM using float16 precision.
  • CodeGemma 7B: Requires approximately 9 GB of memory when loaded in 4-bit mode.

Cloud and Inference Endpoints

  • Google Cloud: CodeGemma can be deployed via Vertex AI or Google Kubernetes Engine (GKE) using Text Generation Inference (TGI).
  • Hugging Face Inference Endpoints: Models can be deployed using TGI as the backend. Note that T4 GPUs do not support the bfloat16 format and require alternative GPU options.

Precision Recommendations

The original checkpoints are released in bfloat16. While float16 is suitable for use and faster on certain hardware, bfloat16 is recommended for maximum precision over float32.

Sources