PrismML-Eng/Bonsai-demo
Bonsai Demo
What it solves
Bonsai provides a family of highly compressed language models (1-bit and ternary) that allow large-scale AI capabilities—including vision, reasoning, and tool calling—to run locally on consumer hardware, such as iPhones or laptops, without requiring massive VRAM or memory offloading.
How it works
The project utilizes extreme quantization (1-bit and ternary weights) to reduce the model's memory footprint. It integrates with llama.cpp and MLX for hardware-accelerated inference across macOS (Metal), Linux, and Windows (CUDA, Vulkan, ROCm). The 27B model specifically incorporates a vision projector for multimodal input and a reasoning mechanism that allows users to budget "thinking" effort per request.
Who it’s for
Developers and AI enthusiasts who want to run powerful, large-context (up to 256k+ tokens) vision-language models locally on limited hardware, as well as those interested in testing extreme quantization formats.
Highlights
- Extreme Compression: 1-bit and ternary model families available in sizes from 1.7B to 27B parameters.
- Multimodal Capabilities: The 27B model supports images, screenshots, and PDFs.
- Agentic Features: Native OpenAI-style tool calling and MCP server integration.
- Reasoning Model: Built-in "thinking" capabilities with adjustable reasoning budgets.
- Broad Hardware Support: Compatible with Apple Silicon, NVIDIA GPUs, and various CPU/GPU backends via llama.cpp.
- Massive Context: Supports conversations up to 262,144 tokens.