PrismML-Eng/Bonsai-demo

What it solves

Bonsai-demo provides a streamlined way to run highly compressed, high-performance language models locally across various hardware (Mac, Linux, Windows) and backends (Metal, CUDA, Vulkan, ROCm, CPU). It specifically addresses the challenge of running large models on consumer hardware by providing 1-bit and ternary (2-bit) quantized versions of the Bonsai model family.

How it works

The project uses a combination of GGUF (via llama.cpp) and MLX formats to deploy models in sizes ranging from 1.7B to 27B parameters. The 27B models are vision-language models capable of processing images, screenshots, and PDFs. The system leverages specialized quantization (1-bit and ternary) to reduce the memory footprint—for example, the 1-bit Bonsai-27B can fit on a modern iPhone. It includes setup scripts that automate dependency installation, model downloading, and the configuration of a chat server with a UI (Open WebUI) that supports agentic tool calling and reasoning effort settings.

Who it’s for

• Local LLM enthusiasts wanting to run large models on limited VRAM or consumer-grade hardware.
• Developers looking to integrate vision-language capabilities and agentic tool calling into local workflows.
• Mac users seeking optimized performance via MLX.

Highlights

• Extreme Quantization: Offers 1-bit and Ternary-Bonsai (2-bit) versions to minimize memory usage.
• Multimodal Capabilities: The 27B model supports vision (images/PDFs) and long context (up to 256k+ tokens).
• Agentic Features: Native OpenAI-style tool calling and MCP server integration.
• Reasoning Control: A "thinking" model where users can adjust reasoning effort (Off to Max) per chat.
• Speculative Decoding: Optional dspark drafter support for faster decoding on compatible hardware.
• Broad Hardware Support: Works across macOS, Linux, and Windows using various GPU backends.