Gemma 3n Release Notes: Multimodal On-Device AI

Gemma 3n is a natively multimodal model designed for local hardware execution, supporting text, image, audio, and video inputs. Its release provides the open-source community with high-performance, small-footprint models capable of running on consumer-grade GPUs and mobile devices.

Model Variants and Memory Efficiency

Gemma 3n is released in two primary sizes, each available as a base and an instruct variant. The models use a non-standard nomenclature where "E" stands for "Effective" parameters, indicating that while the actual parameter counts are higher, memory efficiency improvements allow them to operate with significantly lower VRAM requirements.

  • gemma-3n-E2B: Actual parameter count of 5B, but requires only 2GB of GPU RAM.
  • gemma-3n-E4B: Actual parameter count of 8B, but requires only 3GB of GPU RAM.

Technical Architecture

Gemma 3n integrates a language decoder with specialized encoders for vision and audio to achieve native multimodality.

Vision and Audio Encoders

  • Vision Encoder (MobileNet-V5): Uses MobileNet-v5-300 with 300M parameters. It supports resolutions of 256x256, 512x512, and 768x768. On Google Pixel devices, it achieves 60 FPS, outperforming ViT Giant while using three times fewer parameters.
  • Audio Encoder: Based on the Universal Speech Model (USM), it processes audio in 160ms chunks and supports speech-to-text and translation (e.g., English to Spanish or French).

Architectural Innovations

  • MatFormer Architecture: A nested transformer design that allows subsets of layers to be extracted as individual models. The E2B model is configured as a sub-model of E4B, enabling users to mix and match layers based on their specific memory budget and hardware.
  • Per-Layer Embeddings (PLE): This technique reduces accelerator memory usage by offloading embeddings to the CPU, which allows the 5B parameter E2B model to occupy the VRAM of a 2B parameter model.
  • KV Cache Sharing: This feature accelerates prefill for audio and video processing, resulting in 2x faster prefill compared to Gemma 3 4B.

Performance and Capabilities

Gemma 3n demonstrates high efficiency and broad linguistic reach:

  • LMArena Score: The E4B model is the first sub-10B model to achieve a score of 1300+.
  • Multilingual Support: The model supports 140 languages for text and 35 languages for multimodal interactions.
  • Benchmarks: The model shows competitive performance on MMLU across E4B, E2B, and various Mix-n-Match configurations.

Ecosystem Integration and Deployment

Gemma 3n is integrated across several major open-source libraries and runtimes:

  • Transformers & Timm: The reference implementation for the architecture and the MobileNet-v5 vision encoder.
  • MLX: Day 0 support for all three modalities (text, image, audio).
  • llama.cpp & Ollama: Support for text-only inputs.
  • Transformers.js & ONNXRuntime: ONNX weights are available for the gemma-3n-E2B-it variant, with integration into Transformers.js version 3.6.0.
  • Google AI Edge: Supported for on-device deployment.

Fine-Tuning and Resources

Developers can adapt Gemma 3n for specific tasks using the Hugging Face Gemma Recipes repository, which contains scripts and notebooks. Specific Google Colab notebooks are provided for general fine-tuning on T4 GPUs and specialized fine-tuning for audio tasks.

Sources