PrismML-Eng/Bonsai-demo

Bonsai Demo

What is Bonsai‑demo?

Bonsai‑demo is a ready‑to‑run local demo for the Bonsai 2 family of large language models (LLMs) released by PrismML. It lets you run 2 B, 4 B, 9 B, or the flagship 27 B model on a personal computer – on macOS (Metal), Linux, Windows, or even CPU‑only – and interact with it through a web UI, a simple chat‑server, or the command line.

Key capabilities

Feature How it works
Model sizes 2 B, 4 B, 9 B, 27 B (default). All are ternary‑quantised (≈1.75 bits/weight) and available in two formats: GGUF (for llama.cpp) and MLX‑2‑bit (for Apple‑Silicon).
Vision Upload images, PDFs or screenshots; the model’s vision projector processes them and you can ask questions about the content.
Tool calling OpenAI‑style tool_calls are supported. The UI includes an MCP client with pre‑wired Hugging Face and DeepWiki tools, and you can add your own.
Reasoning (“thinking”) A built‑in “reasoning effort” budget lets the model think before answering. UI lets you pick Off, Low, Medium, High, Max; the default is xhigh for the 27 B model.
Long context Up to 262 K tokens of context via a hybrid‑attention backbone. Optional 4‑bit KV‑cache reduces memory to ~18 KB/token.
Cross‑platform back‑ends Uses llama.cpp (Metal, CUDA, Vulkan, ROCm, CPU) for GGUF checkpoints, and MLX for the 2‑bit checkpoints on Apple Silicon.
Agentic demo A pre‑packaged “Hermes” agent can generate a 3‑D skateboard game from a two‑sentence brief, showing end‑to‑end tool use.
Open WebUI integration Optional ChatGPT‑like UI with image support, tool calling, and a server‑side code interpreter (Jupyter, matplotlib, pandas, etc.).

How to get started (quickest path)

  1. Clone & run setup – git clone https://github.com/PrismML-Eng/Bonsai-demo.git && cd Bonsai-demo && ./setup.sh – the script installs system deps, a Python virtual‑env, downloads the 27 B PQ2_0 checkpoint (≈7.8 GB) and the matching llama.cpp binaries.
  2. Start the chat server – ./scripts/start_llama_server.sh (or the PowerShell equivalent). Open http://localhost:8080 in a browser.
  3. Optional – run a single prompt from the terminal: ./scripts/run_llama.sh -p "What is the capital of France?".

Switching model size or backend

  • Set BONSAI_MODEL=4B (or 2B/9B) before running setup.sh.
  • On Apple Silicon you can use the MLX backend: ./scripts/run_mlx.sh … or ./scripts/start_mlx_server.sh.
  • To force CPU‑only inference, set BONSAI_NGL=0.

Configuration knobs you’ll most likely touch

Variable Default Typical use
BONSAI_THINKING 1 (enabled) Turn off reasoning for speed (0).
BONSAI_CTX auto (RAM‑tiered) Override context length, e.g. BONSAI_CTX=131072.
BONSAI_KV4 0 Enable 4‑bit KV cache for huge contexts (1).
BONSAI_OPENWEBUI / BONSAI_CODE_INTERPRETER 1 Set to 0 to skip the heavy Open WebUI or code‑interpreter install.

What the repository contains

  • Scripts for setup, building llama.cpp from source, launching servers (llama.cpp or MLX), and one‑off prompt runs.
  • Documentation (README, BACKEND‑SUPPORT.md, MODEL‑FORMATS.md, etc.) covering model formats, environment variables, benchmarking, vision, tool‑calling, KV‑cache, and more.
  • Docker‑style assets – a small assets folder with the Bonsai logo.
  • Optional integrations – Open WebUI, Jupyter‑based code interpreter, and a pre‑seeded agentic demo.

Who is this for?

  • Developers or researchers who want to experiment with a 27 B‑scale LLM on‑device without paying for cloud inference.
  • Tool‑builders looking for a local model that already supports OpenAI‑style tool calls and vision.
  • Enthusiasts who want a plug‑and‑play chat UI (Open WebUI) plus the ability to run the model on a laptop, desktop, or Apple‑Silicon Mac.

Bottom line: Bonsai‑demo is a full‑featured, open‑source demo that makes the PrismML “Bonsai 2” LLM family runnable locally with vision, tool calling, long‑context, and optional agentic UI—all orchestrated by easy‑to‑use scripts.

Related