Google Gemma Open LLM Release

Google has released Gemma, a family of open-access large language models (LLMs) based on the Gemini architecture. This release provides the community with high-performance models in 2B and 7B parameter sizes, designed for efficient deployment across consumer-grade GPUs, TPUs, and CPUs.

Model Variants and Specifications

Gemma is available in four primary configurations, each featuring a context length of 8K tokens and the ability to run on consumer hardware without mandatory quantization:

  • gemma-7b: The base pretrained 7B parameter model.
  • gemma-7b-it: The instruction-tuned version of the 7B model.
  • gemma-2b: The base pretrained 2B parameter model.
  • gemma-2b-it: The instruction-tuned version of the 2B model.

Additionally, Google released updated instruction models (gemma-1.1-7b-it and gemma-1.1-2b-it) one month after the initial launch. These 1.1 versions offer improvements in factuality, instruction following, multi-turn conversation quality, coding capabilities, and a reduced tendency to start responses with "Sure,".

Performance Benchmarks

Gemma 7B demonstrates strong performance relative to other open models. According to the LLM Leaderboard, Gemma-7B scored 63.75, placing it competitively against Mistral-7B-v0.1 (60.97) and Llama 2 7B (54.32), and trailing Llama 2 70B Chat (67.87).

In contrast, Gemma 2B scored 46.51, which is noted as being lower than some similarly sized capable models such as Phi 2 (61.33).

Technical Implementation and Prompting

Prompt Format

While base models do not require a specific prompt format, the instruction-tuned (Instruct) versions utilize a specific conversation structure that must be exactly reproduced for optimal performance:

<start_of_turn>user
[User Input]<end_of_turn>
<start_of_turn>model
[Model Response]<end_of_turn>

Training Data and Transparency

Technical reports indicate that the base models were trained on web documents, code, and mathematical texts. The data underwent filtering to remove personally identifiable information (PII), CSAM content, and to perform licensing checks. However, the source material notes that extensive details regarding dataset composition, preprocessing, and the specific hyperparameters used for Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) for the instruct models have not been shared.

Ecosystem Integration and Deployment

Hugging Face Transformers

Integration with the transformers library (v4.38+) allows Gemma to leverage safetensors, 4-bit quantization via bitsandbytes, and Parameter-Efficient Fine-Tuning (PEFT). The models are compatible with torch.compile() and CUDA graphs, which can provide up to a 4x speedup during inference.

Hardware Requirements

  • gemma-7b-it: Requires approximately 18 GB of RAM (compatible with NVIDIA 3090 or 4090 GPUs).
  • 4-bit Quantization: Reduces memory requirements to approximately 9 GB, enabling compatibility with a wider range of consumer cards and Google Colab GPUs.

Deployment Options

  • Google Cloud: Gemma can be deployed via Vertex AI or Google Kubernetes Engine (GKE) using Text Generation Inference (TGI).
  • Hugging Face Inference Endpoints: Supports production-ready deployment using TGI as the backend, enabling features like continuous batching and token streaming.
  • JAX/Flax: Weights are available for JAX/Flax users via the flax revision in the model repository.

Fine-Tuning Capabilities

Gemma can be efficiently fine-tuned on consumer-size GPUs using the Hugging Face TRL (Transformer Reinforcement Learning) library. By employing 4-bit quantization and QLoRA (Quantized Low-Rank Adaptation) targeting all attention block linear layers, a 7B model can be trained on a single A10G GPU in approximately 9 hours using the OpenAssistant chat dataset.

Sources