Prism ML Bonsai 2 27B Near-Lossless Ternary Compression Achieves 9× Smaller Footprint

Takeaway

Prism ML released Ternary Bonsai 2 27B, a ternary‑weight (‑1, 0, +1) version of Qwen 3.8 27B that occupies 5.9 GB (1.76 effective bits per weight) and retains 98.2 % of the full‑precision model’s aggregate benchmark score, delivering more than 9× size reduction with comparable capability.

What the release is

  • Base model: Qwen 3.8 27B (state‑of‑the‑art 27 billion‑parameter multimodal model).
  • Compression scheme: Ternary weights with FP16 group‑wise scaling, yielding an effective 1.76 bits/weight.
  • Model size: 5.9 GB on disk and in memory (≈ 1/9th of the 52 GB full‑precision checkpoint).
  • Context window: 262 K tokens.
  • Modalities: Text and image input.
  • License: Apache 2.0.
  • Supported runtimes: CUDA on NVIDIA GPUs, Apple MLX on macOS/iOS, and a custom Prism‑fork of llama.cpp for GGUF files.

Benchmark performance

Across a diverse suite (reasoning, math, coding, instruction following, vision, agentic tool use) Bonsai 2 27B scores 83.9, which is 98.2 % of the full‑precision Qwen 3.8 27B aggregate (85.4). Detailed per‑task numbers:

Capability Bonsai 2 27B Qwen 3.8 27B Qwen 3.6 27B
Agentic & Tool‑Calling (τ²‑bench) 77.57 79.74 80.05
Coding (HumanEval+, LiveCodeBench) 81.58 82.17 82.57
Instruction Following 82.66 81.25 74.53
Knowledge & Reasoning (MMLU‑Redux, GPQA) 83.95 86.66 84.71
Math (AIME, GSM8K, MATH‑500) 96.57 97.06 94.64
Vision (CharXiv, A‑OKVQA, OmniDocBench) 78.59 81.64 79.82
Overall 83.9 85.4 83.6

Figure I: Benchmark scores (thinking mode) show Bonsai 2 27B closely tracks the full‑precision baseline while using a fraction of the memory.

The most important observation is that coding, vision, and long‑horizon agentic tasks retain the bulk of their capability—areas that typically degrade sharply under aggressive quantization.

Intelligence density

Bonsai 2 27B delivers the highest intelligence‑per‑GB among 27 B‑class models. The accompanying density chart (Figure II in the blog) places it far above conventional 8‑bit or 4‑bit quantizations, confirming that the ternary scheme is an outlier in the trade‑off space.

Throughput and energy efficiency

Hardware Tokens / sec (generation) Energy (mWh/token)
NVIDIA RTX 5090 143 —
NVIDIA RTX 4090 — 0.714
Apple M5 Max 46.8 —

On an RTX 4090, Bonsai 2 27B consumes 40 % less energy per token than a full‑precision 8 B model, making it attractive for battery‑constrained devices and continuous‑background assistants.

Real‑world impact

  • Local knowledge work: Enables coding‑assistant loops, private document analysis, and multimodal debugging without cloud round‑trips.
  • Hardware accessibility: Fits comfortably on 16 GB GPUs (e.g., RTX 3060) and Apple silicon with the Prism llama.cpp fork.
  • Economic shift: Reduces memory and power costs, allowing higher model density in data‑center racks and on‑device deployments.

Community feedback (Hacker News comments)

  • Installation tips: Users report that the GGUF model works with Prism’s custom llama.cpp fork. Example command line (macOS, M5 Pro) shows ~20 tok/s, with occasional spikes to 44 tok/s after a restart. (-ngl 99 -fa on -c 32768)
  • Performance variance: Some users see 7‑8 tok/s on a Mac Mini M2 (16 GB RAM) and 0.67 tok/s on a 6 GB GPU, indicating that sufficient VRAM (≈ 8 GB) is still required for smooth operation.
  • Comparison to other quantizations: A comment notes that Q2 quantizations of the same base model (≈ 2.6 bits/weight) sit at the edge of “noticeably worse,” suggesting Bonsai’s ternary approach offers a better quality‑size trade‑off.
  • Browser demo: A community‑built WebGPU demo runs the model entirely in the browser, though users report instability on longer tasks.
  • Speculative decoding: Benchmarks on Radeon RX 7900 XTX with speculative decoding reach ~89 tok/s generation and ~474 tok/s ingestion, with peak VRAM ≈ 10 GiB for a 24 K context.
  • Hardware compatibility questions: Users ask about AMD/HIP support; a workaround is to use the PTQ2_0 variant, which is faster on AMD GPUs but requires slightly more VRAM.
  • Clarifications: The 5.9 GB figure refers to the on‑disk and in‑memory footprint of the ternary weights; runtime memory usage is roughly the same plus overhead for scaling tensors.

How to get started

  1. Download the GGUF model from Hugging Face: prism-ml/Ternary-Bonsai-2-27B-gguf.
  2. Install Prism’s llama.cpp fork (release prism-b10685-7dffb15).
  3. Run the server:
    ./llama-prism-b10685-7dffb15/llama-server \
    -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --port 8331 -ngl 99 -fa on -c 32768
  4. Query via the built‑in web UI or an OpenAI‑compatible API endpoint.

Future directions

Prism ML’s roadmap mentions an 8 B v2 model based on Qwen 3.8, targeting direct phone deployment. Community interest is also high for extending the ternary compression to larger MoE models (e.g., Qwen 3.8‑Flash‑Next) and for integrating the technique into AMD/HIP pipelines.

Bottom line

Ternary Bonsai 2 27B proves that near‑lossless compression is now practical for 27 B‑scale multimodal models, delivering sub‑6 GB footprints, high throughput, and strong performance on the most demanding tasks. This release narrows the gap between cutting‑edge research models and the hardware constraints of everyday devices, reshaping how AI will be deployed across the stack.

Sources

Related