Cactus Hybrid: Adding Confidence Scoring to Gemma 4 for Cloud Handoff

Cactus Hybrid: Adding Confidence Scoring to Gemma 4 for Cloud Handoff

Cactus Hybrid enables small, on-device language models to identify when their own answers are likely incorrect, allowing for a seamless handoff to larger cloud-based models. By integrating confidence probes directly into the model checkpoint, the system provides a structured confidence score between 0 and 1 for every response, eliminating the need to parse confidence from the generated text.

Hybrid Routing for Performance and Privacy

Cactus Hybrid uses a routing strategy where an on-device model handles queries when confidence is high and routes to a larger model when confidence falls below a specific threshold (e.g., 0.85). This approach combines the speed and privacy of local execution with the reliability of large-scale models.

Using the Gemma 4 E2B Hybrid model, the system can match the performance of Gemini 3.1 Flash-Lite on various benchmarks while only routing 15–35% of queries to the cloud model. The handoff requirements vary by benchmark and quantization level:

Benchmark Handoff to match Flash-Lite (FP16) At 4-bit At 3-bit
ChartQA 15†20% 25–30% 40–50%
MMBench 30–35% 40–45% 50–55%
LibriSpeech 25–30% 35–40% 55–65%
GigaSpeech 30–35% 40–45% 50–55%
MMAU 30–35% 35–40% 50–55%
MMLU-Pro 45–55% ~90% n/a

Modality-Independent Correctness Signals

Routing quality is measured by the Area Under the Receiver Operating Characteristic (AUROC), which indicates how effectively the model separates correct answers from incorrect ones. The Gemma 4 E2B Hybrid probe achieves a mean AUROC of 0.814, significantly outperforming token entropy (0.549).

Notably, the confidence probe was trained on zero audio data, yet it achieved AUROC scores between 0.79 and 0.88 on audio benchmarks including GigaSpeech and LibriSpeech. This suggests the probe is detecting a modality-independent correctness signal within the model's hidden states rather than memorizing modality-specific patterns.

Hold-out Modality Cactus Hybrid Token Entropy
MMLU text MCQ 0.770 0.697
MMLU-Pro text MCQ 0.771 0.692
ARC-Easy text MCQ 0.888 0.655
ARC-Challenge text MCQ 0.834 0.646
GSM8K (3-shot) text gen 0.782 0.731
MMBench-EN-Dev vision MCQ 0.840 0.435
ChartQA vision QA 0.779 0.615
DocVQA vision QA 0.781 0.512
MMAU audio MCQ 0.789 0.517
GigaSpeech audio 0.876 0.343
Earnings-22 audio 0.839 0.323
LibriSpeech audio 0.822 0.427
Mean 0.814 0.549

Implementation and Integration

Cactus Hybrid provides support for multiple deployment frameworks. A critical implementation detail for Transformers users is the requirement to use an explicit .to(device) call rather than device_map="auto", as the probe scores generations outside the standard module forward() path, and meta-device weights can cause crashes during confidence reads.

Framework Support

  • Cactus: Available via pip install cactus-compute.
  • MLX: Available via Cactus-Compute/gemma-4-e2b-it-hybrid-mlx.
  • Transformers: Requires transformers>=5.5.4,<5.6 and torch (note: version 5.14+ may cause segfaults).
  • llama.cpp: Requires a custom patch compiled into the engine to enable the confidence field in the API response.

Community Insights and Technical Critique

Community discussion highlighted several theoretical and practical considerations regarding the model's confidence scoring:

"You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct."

Technical suggestions from the community included the use of conformal prediction to calibrate the cutoff and provide distribution-free guarantees on the rate of wrongly-kept on-device answers. Other users questioned whether combining token entropy with the probe's confidence score would further improve AUROC results, or if the post-training of the probe affects the general quality of the model's responses.

Sources