Gemma 4 release: open‑source multimodal models with on‑device support

TL;DR

Gemma 4, the new multimodal family from Google DeepMind, is now openly available on Hugging Face under Apache 2.0, offering image, audio, and video understanding, up to 256 K context windows, and on‑device deployment via transformers, llama.cpp, MLX, Rust, and WebGPU.

---

What’s New in Gemma 4?

Gemma 4 adds true multimodality (image, audio, video) to the Gemma line, supports variable‑aspect‑ratio image token budgets, and provides five model sizes—from a 2.3 B‑parameter edge model (E2B) to a 31 B dense model—each with base and instruction‑tuned checkpoints. All models support a 128 k or 256 k token context window, enabling long‑form reasoning and agentic use cases on‑device.

Model	Effective Params	Context	Checkpoints
Gemma 4 E2B	2.3 B (5.1 B with embeddings)	128 k	base, instruction‑tuned
Gemma 4 E4B	4.5 B (8 B with embeddings)	128 k	base, instruction‑tuned
Gemma 4 12B Unified	11.95 B dense (encoder‑free)	256 k	base, instruction‑tuned
Gemma 4 31B	31 B dense	256 k	base, instruction‑tuned
Gemma 4 26B A4B (MoE)	26 B total, 4 B active	256 k	base, instruction‑tuned

Architecture Highlights

Alternating Sliding‑Window and Global Attention

Smaller dense models use 512‑token sliding windows; larger models use 1024‑token windows, alternating with full‑context layers to keep compute linear while preserving long‑range dependencies.

Dual RoPE Configurations

Standard rotary positional embeddings (RoPE) power sliding layers; a pruned RoPE variant powers global layers, extending the effective context without extra memory.

Per‑Layer Embeddings (PLE)

PLE adds a lightweight, per‑layer conditioning vector for each token. It combines a token‑identity lookup with a context‑aware projection, allowing later layers to receive token‑specific signals without overloading the initial embedding table. This improves specialization at modest parameter cost and works for multimodal placeholders by using the pad token ID.

Shared KV Cache

The last N layers reuse key‑value (KV) tensors from the preceding non‑shared layer of the same attention type. This reduces both compute and memory during inference, especially for long‑context generation, with negligible quality loss.

Vision and Audio Encoders

• Vision encoder (all models except 12B) uses learned 2‑D positions, multidimensional RoPE, and supports token budgets of 70‑1120 tokens, preserving original aspect ratios.
• Audio encoder (E2B, E4B) is a USM‑style conformer identical to Gemma‑3n.
• Unified 12B drops separate encoders; raw image patches and audio waveforms are projected directly into the LLM embedding space, simplifying latency and fine‑tuning.

Multimodal Capabilities

Gemma 4 handles OCR, speech‑to‑text, object detection, pointing, and multimodal function calling out‑of‑the‑box. The model follows a strict input ordering: images before text, audio after text. This convention is required for the built‑in chat template.

Example: GUI Element Detection

Prompt: “What’s the bounding box for the "view recipe" element in the image?” The model returns JSON with coordinates relative to a 1000 × 1000 canvas, requiring no post‑processing.

Example: Video Understanding

Gemma 4 can ingest video URLs with optional audio extraction. Smaller models (E2B/E4B) can process audio; larger models handle video‑only streams. Sample outputs demonstrate accurate scene description and song‑topic inference.

Example: Audio Question Answering & Transcription

Using the same chat template, Gemma 4 can answer detailed questions about speech or produce verbatim transcriptions. The E4B model’s transcription matches the reference text with punctuation and capitalization.

Example: Multimodal Function Calling

When asked “What is the city in this image? Check the weather there,” the model identifies the city (Bangkok) and emits a correctly formatted tool call get_weather(city="Bangkok").

Deploy Anywhere

Gemma 4 ships with day‑0 support for a wide range of inference engines, enabling edge and browser deployment.

Transformers (Python)

pip install -U transformers
from transformers import pipeline
pipe = pipeline("any-to-any", model="google/gemma-4-e2b-it")

The pipeline accepts mixed‑modality messages and optional load_audio_from_video=True for video.

Llama.cpp (C++)

curl -LsSf https://llama.app/install.sh | sh
llama serve -hf ggml-org/gemma-4-E2B-it-GGUF

GGUF checkpoints are provided for all sizes; quantization can be chosen at serve time.

Transformers.js (Browser)

Gemma 4 runs in WebGPU via transformers.js. A demo space showcases image‑text and audio‑text inference directly in the browser.

MLX (Apple Silicon)

The mlx‑vlm library offers TurboQuant 4× memory reduction and supports long‑context inference on M‑series chips.

Mistral.rs (Rust)

Mistral.rs provides an OpenAI‑compatible server with built‑in tool calling and multimodal support. Install with a single script and serve any Gemma 4 checkpoint.

Multi‑Token Prediction Drafters

Google released speculative drafters for all Gemma 4 sizes. The drafter proposes several future tokens in a single forward pass; the target model validates them, yielding up to ~3× speed‑up without quality loss. Drafters share the KV cache with the main model and use embedder clustering for the edge variants.

DiffusionGemma: Text Generation via Diffusion

DiffusionGemma (26B A4B) generates text by denoising blocks of tokens in parallel rather than token‑by‑token autoregression. It achieves 15–20 tokens per forward pass, exceeding 1100 tokens/s on H100 FP8 for low batch sizes. Accuracy drops modestly (e.g., MMLU Pro 77.6 % vs 82.6 % for the autoregressive 26B A4B) but the throughput gain is significant for latency‑sensitive applications.

Fine‑Tuning Made Easy

TRL Integration

Gemma 4 works with the trl library for supervised fine‑tuning, including multimodal tool‑response training. An example script trains the model to drive in the CARLA simulator, demonstrating vision‑to‑action learning.

Vertex AI Support

A full Vertex AI example shows how to launch a custom container with CUDA, install Transformers and TRL, and fine‑tune Gemma 4 on an H100 instance.

Unsloth Studio

Unsloth Studio provides a UI for local or Colab‑based fine‑tuning. Users can select any Gemma 4 checkpoint and start training with a few clicks.

Benchmark Results

Gemma 4 establishes a new Pareto frontier across reasoning, coding, vision, audio, and long‑context tasks. Highlights (instruction‑tuned models):

Benchmark	31B	26B A4B	12B Unified	4B E4B	2.3B E2B
MMLU Pro	85.2 %	82.6 %	77.2 %	69.4 %	60.0 %
AIME 2026 (no tools)	89.2 %	88.3 %	77.5 %	42.5 %	37.5 %
GPQA Diamond	84.3 %	82.3 %	78.8 %	58.6 %	43.4 %
LiveCodeBench v6	80.0 %	77.1 %	72.0 %	52.0 %	44.0 %
MMMU Pro (vision)	76.9 %	73.8 %	69.1 %	52.6 %	44.2 %
MRCR v2 128k (long context)	66.4 %	44.1 %	43.4 %	25.4 %	19.1 %

The 31 B dense model achieves an estimated LMArena text‑only score of 1452, while the 26 B MoE reaches 1441 with only 4 B active parameters, demonstrating extreme efficiency.

Implications

• Open‑source frontier – Fully Apache 2.0‑licensed, Gemma 4 gives the community a high‑quality multimodal model that can be run on phones, laptops, and servers without licensing barriers.
• On‑device agents – The combination of long context, shared KV cache, and efficient quantization makes Gemma 4 ideal for local assistants, autonomous robotics, and privacy‑preserving applications.
• Tool‑calling parity – Multimodal function calling works the same way as text‑only calls, enabling richer agentic workflows (e.g., vision‑guided weather queries).
• Speed‑vs‑quality trade‑offs – DiffusionGemma offers a new speed‑first path for generation‑heavy workloads, while MTP drafters accelerate standard Gemma 4 inference without sacrificing answer quality.

---

Try Gemma 4 Today

• Transformers demos – Interactive spaces for E4B, 12B Unified, 26B A4B, and 31B.
• WebGPU demo – Run Gemma 4 in the browser via transformers.js.
• Model hub – All checkpoints, GGUF files, and ONNX exports are available at https://huggingface.co/collections/google/gemma-4.

---

Acknowledgements

The release was made possible by Google DeepMind’s model contributions and extensive integration work by the Hugging Face community: Cyril, Raushan, Eustache, Arthur, Lysandre (transformers), Joshua (transformers.js), Eric (mistral.rs), Son (llama.cpp), Prince (MLX), Quentin, Albert, Kashif (TRL), Adarsh (SGLang), and Toshihiro (demo engineering).