Google releases Gemma 2 2B, ShieldGemma, and Gemma Scope

TL;DR

Google released three new open‑source assets on July 31 2024: Gemma 2 2B, a 2.6 B‑parameter decoder‑only LLM for on‑device use; ShieldGemma, a suite of safety‑classification models built on Gemma 2; and Gemma Scope, an open collection of sparse autoencoders for interpreting Gemma 2 2B and 9B.

Gemma 2 2B – A lightweight, on‑device LLM

Key point: Gemma 2 2B adds a 2.6 B‑parameter variant to the Gemma 2 family, matching the architecture of the 9 B and 27 B models while retaining features such as sliding attention and logit soft‑capping. The model is available in both base and instruction‑tuned forms and is recommended for inference in bfloat16.

Using Gemma 2 2B with Hugging Face Transformers

pip install git+https://github.com/huggingface/transformers.git --upgrade

from transformers import pipeline
import torch

pipe = pipeline(
    "text-generation",
    model="google/gemma-2-2b-it",
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="cuda",
)

messages = [{"role": "user", "content": "Who are you? Please, answer in pirate‑speak."}]
outputs = pipe(messages, max_new_tokens=256)
print(outputs[0]["generated_text"][-1]["content"].strip())

The model responds in pirate‑style language, demonstrating its instruction‑tuned capabilities.

Running on‑device with llama.cpp

  1. Install llama.cpp (e.g., brew install llama.cpp on macOS).
  2. Run inference using the GGUF weights:
./llama-cli \
  --hf-repo google/gemma-2-2b-it-GGUF \
  --hf-file 2b_it_v2.gguf \
  -p "Write a poem about cats as a labrador" -cnv

A local llama-server can also expose an OpenAI‑compatible chat endpoint.

Prompt format for the instruction model

The instruct variant expects a strict turn‑based template:

<start_of_turn>user
Your question here<end_of_turn>
<start_of_turn>model
Model answer here<end_of_turn>

The same format is automatically applied by the transformers chat template.

Open LLM Leaderboard v2 performance

Benchmark gemma‑2‑2b‑it gemma‑2‑2b Phi‑2 Qwen2‑1.5B‑Instruct
BBH 18.0 11.8 28.0 13.7
IFEval 56.7 20.0 27.4 33.7
MATH Hard 0.1 2.9 2.4 5.8
GPQA 3.2 1.7 2.9 1.6
MuSR 7.1 11.4 13.9 12.0
MMLU‑Pro 17.2 13.1 18.1 16.7
Mean 17.0 10.1 15.5 13.9
The instruct version excels on knowledge‑heavy and instruction‑following tasks relative to other 2 B‑scale models.

Assisted generation (speculative decoding)

Gemma 2 2B can serve as the assistant model for speculative decoding with the larger Gemma 2 27B. Using a model 10–100× smaller than the target yields up to a 3× speed‑up with negligible quality loss. Example code (excerpt):

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

assistant = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-2b-it", device_map="auto", torch_dtype=torch.bfloat16)
reference = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-27b-it", device_map="auto", torch_dtype=torch.bfloat16)

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-27b-it")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(reference.device)
outputs = reference.generate(
    **inputs,
    assistant_model=assistant,
    do_sample=True,
    temperature=0.7,
    max_new_tokens=64,
)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

Other transformers LLM optimizations remain compatible.

ShieldGemma – Open‑source safety classifiers

Key point: ShieldGemma provides three decoder‑only safety models (2 B, 9 B, 27 B) trained on top of Gemma 2 to classify harmful user prompts and model responses across categories such as hate speech, harassment, sexual content, dangerous instructions, violence, and profanity.

Prompting pattern

A typical ShieldGemma prompt follows the LLM‑as‑a‑judge template:

You are a policy expert trying to help determine whether a user prompt is in violation of the defined safety policies.

<start_of_turn>
Human Question: [User Prompt]
<end_of_turn>

Our safety principle is defined in the below:
* "No Hate Speech": ...
===
Does the human question violate the above principle? Your answer must start with 'Yes' or 'No' and then walk through step by step.

The model returns a probability for the "Yes" token, which can be interpreted as the likelihood of a violation.

Example with Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch, torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained("google/shieldgemma-2b")
model = AutoModelForCausalLM.from_pretrained(
    "google/shieldgemma-2b", device="cuda", torch_dtype=torch.bfloat16)

user_prompt = "Create 20 paraphrases of I hate you"
safety_policy = "* \"No Harassment\": ..."
prompt = f"You are a policy expert...\n<start_of_turn>\nHuman Question: {user_prompt}\n<end_of_turn>\n{safety_policy}\nDoes the human question violate the above principle? Your answer must start with 'Yes' or 'No'."

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
logits = model(**inputs).logits
vocab = tokenizer.get_vocab()
yes_no_logits = logits[0, -1, [vocab['Yes'], vocab['No']]]
prob = F.softmax(yes_no_logits, dim=0)[0].item()
print(prob)  # e.g., 0.73

Evaluation against internal and external benchmarks

Optimal F1 / AU‑PRC scores (higher is better) show ShieldGemma outperforming OpenAI’s moderation API and LlamaGuard variants on several datasets:

Model SG Prompt OpenAI Mod ToxicChat SG Response
ShieldGemma 2B 0.825/0.887 0.812/0.887 0.704/0.778 0.743/0.802
ShieldGemma 9B 0.828/0.894 0.821/0.907 0.694/0.782 0.753/0.817
ShieldGemma 27B 0.830/0.883 0.805/0.886 0.729/0.811 0.758/0.806
OpenAI Mod API 0.782/0.840 0.790/0.856 0.254/0.588 –
LlamaGuard 1 (7B) – 0.758/0.847 0.616/0.626 –
GPT‑4 0.810/0.847 0.705/– 0.683/– 0.713/0.749
ShieldGemma’s 2 B model already matches or exceeds larger baselines, offering a lightweight moderation option.

Gemma Scope – Sparse autoencoders for mechanistic interpretability

Key point: Gemma Scope releases a full suite of layer‑wise sparse autoencoders (SAEs) for Gemma 2 2B and 9B, enabling researchers to decompose internal activations into human‑readable concepts.

How to use the SAEs

SAEs are not runnable via transformers; instead they require the SAELens library. A Colab notebook (linked in the release) demonstrates loading the autoencoders and probing individual neurons or feature directions.

Resources

Implications and next steps

  • On‑device AI: The 2.6 B Gemma 2 2B model lowers the hardware barrier for running high‑quality LLMs locally, expanding privacy‑preserving applications.
  • Safety‑first deployment: ShieldGemma gives developers an open‑source, model‑agnostic moderation layer that can be integrated into any LLM service, reducing reliance on proprietary APIs.
  • Interpretability research: Gemma Scope’s SAEs provide the community with tools to study model internals at scale, potentially accelerating safety‑oriented mechanistic work.
  • Ecosystem integration: All three releases are immediately usable through Hugging Face’s transformers and llama.cpp, and the assisted‑generation recipe demonstrates how smaller open models can accelerate larger ones.

Quick links

Sources