Gemma 3 Release Notes / What's New

Google has released Gemma 3, a new generation of open-weight large language models (LLMs) that introduces native multimodality, expanded multilingual support, and significantly larger context windows. The model family includes four sizes—1B, 4B, 12B, and 27B parameters—offering both pre-trained (base) and instruction-tuned (IT) versions.

Core Capabilities and Model Variants

Gemma 3 provides a scalable range of models designed for different deployment environments, from on-device applications to high-performance servers. While the 1B model remains text-only, the 4B, 12B, and 27B variants are fully multimodal, capable of processing both images and text.

Model Pre-trained Instruction Tuned Multimodal Multilingual Context Window
1B gemma-3-1b-pt gemma-3-1b-it No English 32K
4B gemma-3-4b-pt gemma-3-4b-it Yes 140+ Languages 128K
12B gemma-3-12b-pt gemma-3-12b-it Yes 140+ Languages 128K
27B gemma-3-27b-pt gemma-3-27b-it Yes 140+ Languages 128K

Technical Enhancements

Gemma 3 introduces three primary technical upgrades over Gemma 2 to improve versatility and efficiency.

Expanded Context Length

Context windows have been increased from 8k in Gemma 2 to 128k for the 4B, 12B, and 27B models (and 32k for the 1B model). To achieve this without retraining from scratch, Google used the following methods:

  • Positional Embedding Adjustments: RoPE base frequency was upgraded from 10k to 1M and scaled by a factor of 8.
  • KV Cache Optimization: The models utilize sliding window interleaved attention. Hyperparameters were tuned to interleave five local layers with one global layer and reduce the window size to 1024 tokens.

Native Multimodality

Multimodal capabilities are powered by the SigLIP image encoder, which processes square images resized to 896x896. To handle non-square aspect ratios and high-resolution images during inference, Gemma 3 employs a "pan and scan" algorithm that adaptively crops images to zoom in on details.

Attention mechanisms differ based on the input type:

  • Text: Uses one-way (causal) attention.
  • Images: Uses full bidirectional attention, allowing the model to analyze all parts of an image without masks.

Multilingual Support

Language coverage was expanded by doubling the amount of multilingual data in the pretraining dataset. The models utilize the Gemini 2.0 SentencePiece tokenizer with 262K entries, which specifically improves the encoding of Chinese, Japanese, and Korean text.

Performance and Evaluation

In human-preference evaluations on the LMSys Chatbot Arena, the Gemma 3 27B IT model achieved an Elo score of 1339, ranking it among the top 10 models overall and making it comparable to o1-preview.

Benchmark performance for the 27B model includes:

  • Reasoning and Math: 69.0 on MATH and 42.4 on GPQA Diamond.
  • Multimodal and Factual: 64.9 on MMMU and 74.9 on FACTS Grounding.
  • Coding and SQL: 29.7 on LiveCodeBench and 54.4 on Bird-SQL.
  • General Knowledge: 67.5 on MMLU-Pro.

While competitive with closed Gemini models, the 27B model showed a weakness in basic facts, scoring 10.0 on SimpleQA.

Deployment and Inference

Hugging Face Transformers

Gemma 3 is integrated into the transformers library via two primary classes:

  • Gemma3ForConditionalGeneration: Used for the 4B, 12B, and 27B vision-language models.
  • Gemma3ForCausalLM: Used for the 1B text-only model or to run multimodal models as text-only LLMs by omitting the vision tower.

On-Device and Low-Resource Options

For local deployment, Gemma 3 supports several frameworks:

  • MLX: Day-zero support via mlx-vlm for Apple Silicon (Macs and iPhones).
  • Llama.cpp: Pre-quantized GGUF files are available for local CPU/GPU execution.
  • Hugging Face Endpoints: One-click deployment is available for the 12B and 27B IT models via the Inference Catalog.

Sources