Cactus Hybrid Gemma 4: On-Device LLMs with Confidence-Based Cloud Handoff

Cactus Hybrid allows on-device models to detect when they are likely to be incorrect, providing a confidence score between 0 and 1 for every response. This enables a "hybrid" architecture where a small, fast, and private local model handles high-confidence queries, while low-confidence queries are automatically routed to a larger cloud-based model to maintain accuracy.

High-Accuracy Routing with Low Cloud Overhead

By routing only a fraction of queries to a larger model, Cactus Hybrid can match the performance of larger models while maintaining the speed of local execution. The Gemma 4 E2B Hybrid model matches Gemini 3.1 Flash-Lite on most benchmarks by routing only 15–35% of queries to the cloud model.

Performance by Quantization Level

Confidence-based handoff allows the local model to maintain parity with Gemini 3.1 Flash-Lite (FP16) across various benchmarks, though the required handoff rate increases as quantization becomes more aggressive:

Benchmark	Handoff to match Flash-Lite (FP16)	At 4-bit	At 3-bit
ChartQA	15–20%	25–30%	40–50%
MMBench	30–35%	40–45%	50–55%
LibriSpeech	25–30%	35‐40%	55–65%
GigaSpeech	30–35%	40–45%	50–55%
MMAU	30–35%	35‐40%	50–55%
MMLU-Pro	45–55%	~90%	n/a

Modality-Independent Correctness Signals

Cactus Hybrid uses a probe trained to read hidden states from the model's layers to predict correctness. This approach significantly outperforms traditional token entropy for determining if an answer is correct.

Across a variety of modalities, the Gemma 4 E2B Hybrid achieves a mean AUROC (Area Under the Receiver Operating Characteristic curve) of 0.814, compared to 0.549 for token entropy. Notably, the probe was trained on zero audio data, yet it achieved an AUROC between 0.79 and 0.88 on four different audio benchmarks (including GigaSpeech and LibriSpeech). This indicates that the probe is detecting a modality-independent signal of correctness within the hidden states rather than memorizing specific patterns.

AUROC Comparison: Cactus Hybrid vs. Token Entropy

Hold-out	Modality	Cactus Hybrid	Token Entropy
MMLU	text MCQ	0.770	0.697
MMLU-Pro	text MCQ	0.771	0.692
ARC-Easy	text MCQ	0.888	0.655
ARC-Challenge	text MCQ	0.834	0.646
GSM8K (3-shot)	text gen	0.782	0.731
MMBench-EN-Dev	vision MCQ	0.840	0.435
ChartQA	vision QA	0.814	0.615
DocVQA	vision QA	0.781	0.512
MMAU	audio MCQ	0.789	0.517
GigaSpeech	audio	0.876	0.343
Earnings-22	audio	0.839	0.323
LibriSpeech	audio	0.822	0.427
Mean		0.814	0.549

Implementation and Integration

Cactus Hybrid provides support for multiple runtimes to facilitate on-device deployment:

Cactus Runtime

Using the cactus-compute library, developers can initialize the model and receive a response along with a structured confidence score.

MLX

For Apple Silicon, the mlx-lm library can be used. The model reasons before answering, and the confidence score is accessible via model.last_confidence.

Transformers

Integrated with the transformers library (versions 5.5.4 to 5.6), the model can be returned with return_confidence=True during generation.

llama.cpp

Because llama.cpp is written in C++, the probe is implemented as a patch that must be compiled into the engine. Once patched, the llama-server provides a top-level confidence field in its JSON response.

Community Insights and Technical Discussion

Technical discussions surrounding the release have highlighted several considerations for confidence scoring in LLMs:

"You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct."

Critics have suggested that the model is not predicting "correctness" per se, but rather its own internal uncertainty or inconsistency. Additionally, some developers have proposed using conformal prediction to calibrate the cutoff thresholds to provide guaranteed bounds on the rate of wrongly-kept on-device answers.