Bonsai 27B Brings 27‑Billion‑Parameter AI to Phones and Laptops
Bonsai 27B Brings 27‑Billion‑Parameter AI to Phones and Laptops
Bonsai 27B puts a 27‑billion‑parameter model on a phone for the first time
Takeaway: PrismML’s Bonsai 27B compresses the Qwen‑3.6 27B model to 3.9 GB (1‑bit) and 5.9 GB (ternary) while keeping 90‑95 % of the original accuracy, enabling on‑device multimodal reasoning, tool use, and agentic loops on modern smartphones and laptops.
Two low‑bit operating points cover phones and laptops
Conclusion: The 1‑bit variant fits the memory budget of an iPhone 17 Pro (≈4 GB usable) and the ternary variant runs comfortably on a typical laptop (≈6 GB usable). Both models retain the full reasoning pipeline—embeddings, attention, MLPs, and LM head—without any higher‑precision escape.
- 1‑bit Bonsai 27B – binary {‑1,+1} weights with group‑wise FP16 scaling, 1.125 effective bits/weight, 3.9 GB total size. Designed for phone‑class footprints.
- Ternary Bonsai 27B – ternary {‑1,0,+1} weights with the same scaling, 1.71 effective bits/weight, 5.9 GB total size. Optimized for laptop‑class quality.
Both variants ship a compact 4‑bit vision tower and a 262K‑token context window, and they support speculative decoding for lossless draft‑and‑verify speedups.
Benchmark results show minimal loss of intelligence
Conclusion: Across a 15‑benchmark suite, Bonsai 27B retains the majority of the full‑precision Qwen 3.6 27B performance, with the biggest drops in vision and tool‑calling but still within a few points of the baseline.
| Category | Full‑precision Qwen 3.6 27B | Ternary Bonsai 27B | 1‑bit Bonsai 27B |
|---|---|---|---|
| Math (GSM8K, MATH‑500, AIME) | 95.3 | 93.4 | 91.7 |
| Coding (HumanEval+, MBPP+, LiveCodeBench) | 88.7 | 86.0 | 81.9 |
| Agentic & Tool‑calling (BFCL v3, TauBench) | 80.0 | 74.0 | 66.0 |
| Instruction following (IFEval, IFBench) | 78.4 | 71.8 | 65.8 |
| Knowledge / STEM (MMLU‑Redux, MuSR) | 83.1 | 77.0 | 73.4 |
| Vision (MMMU Pro, OCRBench) | 72.6 | 65.2 | 59.6 |
| Overall (15 benchmarks) | 85.0 | 80.5 | 76.1 |
Figure I: Bonsai 27B (thinking mode) retains 95 % of full‑precision performance for ternary and 90 % for 1‑bit across the benchmark suite.
The authors note that math and coding scores are “nearly untouched,” and tool‑calling stays within a few points of the baseline—critical for agentic workloads.
Intelligence density breaks the 27B barrier
Conclusion: Bonsai 27B achieves an intelligence‑per‑GB metric of 0.53 / GB, more than ten times the density of a full‑precision 27B model and roughly 2.7 × the best competing low‑bit alternative.
Figure II: Shows Bonsai 27B’s superior intelligence density compared to other 27B‑class models.
The shift means a model with the full capability set of modern LLMs now fits on devices that previously could only host 2‑B‑parameter models.
Why on‑device agentic AI matters
Conclusion: Local execution eliminates per‑token API costs, removes latency from round‑trip network calls, and guarantees that private data never leaves the device, opening up offline assistants, persistent on‑device agents, and hybrid architectures that offload only the hardest steps to the cloud.
- Agentic loops often require hundreds of model calls; cloud‑only execution incurs cumulative latency and cost.
- A 12 GB iPhone typically offers ~6 GB usable memory; Bonsai 1‑bit’s 4 GB footprint leaves room for KV‑cache and activations.
- Hybrid deployments can route privacy‑sensitive tasks to the on‑device model while reserving cloud models for frontier queries.
Performance on modern hardware
Conclusion: Bonsai 27B delivers competitive token‑per‑second throughput on both high‑end GPUs and Apple silicon.
- NVIDIA RTX 5090: 163 tok/s (1‑bit) / 134 tok/s (ternary)
- Apple M5 Max: 87 tok/s (1‑bit) / 58 tok/s (ternary)
These numbers demonstrate that the compression does not sacrifice speed, especially when paired with speculative decoding.
Community feedback and open questions
Conclusion: Early adopters raise several practical concerns that will shape future iterations.
- Comparison to other quantized models – A top comment asks for a head‑to‑head with Google’s 4‑bit QAT Gemma 4 12B, noting that Gemma runs at ~7 GB and shows strong tool‑use and vision capabilities.
- KV‑cache efficiency – One user highlights the model’s frugal KV‑cache usage, suggesting it could be valuable for multi‑agent coding workflows.
- Implementation hurdles – Several commenters report difficulties loading the models in LM Studio or via llama.cpp, indicating that ecosystem tooling still needs polishing.
- Hardware limits – Questions remain about the largest model that could fit on a 16 GB GPU at the 1.125‑effective‑bit level.
- Business impact – An investor‑focused comment argues that on‑device high‑capacity models could undercut privacy‑focused AI SaaS startups.
These discussions underline the importance of transparent benchmarking, tooling support, and clear documentation of memory‑budget calculations.
Availability and licensing
Conclusion: Bonsai 27B is released under the Apache 2.0 license, with weights hosted on Hugging Face and a limited‑time developer preview API for quick experimentation.
- Native support on Apple devices via MLX and on NVIDIA GPUs via custom CUDA kernels.
- Full technical details—including compression algorithms, evaluation methodology, and benchmark scripts—are available in the accompanying whitepaper.
Outlook: intelligence density as a new AI progress axis
Conclusion: By moving the intelligence‑per‑GB frontier leftward, Bonsai 27B expands the set of devices that can run advanced AI, reshaping economics from phones to single‑GPU servers.
PrismML emphasizes that their compression methodology is architecture‑agnostic, suggesting future releases will continue to push larger models into on‑device form factors.
The Bonsai 27B release marks a concrete step toward ubiquitous, privacy‑preserving AI by delivering 27‑billion‑parameter capability on everyday hardware.