Needle 2 14 MB Agentic LLM Brings On‑Device Tool Calling to Sub‑$200 Devices

Needle 2 delivers a full‑session LLM for $200‑class devices

Needle 2 packs a 45 M‑parameter model into a 14 MB binary, runs a complete inference session in 28 MB of RAM, and reaches 500 tokens/sec decode speed on a Raspberry Pi 5. This makes functional, privacy‑preserving AI feasible on cheap phones, wearables, microcontrollers, and robots.

Core technical achievements

1. Lossless 2‑bit quantization enables a 14 MB footprint

Needle 2 is trained end‑to‑end with Cactus Quants, a custom CQ2‑bit format that quantizes weights, activations, and KV cache during pre‑training. Because the model never sees full‑precision weights, the 2‑bit representation incurs no quality loss, allowing 45 M parameters to occupy only 14 MB on disk.

2. Simple Attention Network reduces arithmetic per token

The architecture replaces dense MLP projections with a fixed Walsh‑Hadamard transform and learned diagonal matrices, cutting matmul‑active parameters from 82 M to 35 M. The resulting compute cost is 70 MFLOPs per token, roughly a 2‑3× reduction versus a conventional transformer of similar size.

3. Byte‑level grammar and engram lookup prune the vocabulary

A compiled byte‑level grammar restricts the set of legal tokens at each step, skipping up to 98 % of the softmax for structural tokens. An engram layer reads a few rows from hashed n‑gram tables (≈8 M parameters) without any arithmetic, further lowering FLOPs and memory bandwidth.

4. Fixed‑size sliding KV cache guarantees 28 MB RAM ceiling

A 256‑token sliding window bounds the KV cache, so session memory never grows with conversation length. System prompts and tool schemas are pinned as permanent sinks, ensuring the model never forgets its tool definitions.

5. Single‑binary, hardware‑adaptive runtime

The C++ engine probes the CPU at startup and selects the optimal kernel (SDOT, NEON, AVX2, RISC‑V vectors, WASM SIMD, or scalar). All weights remain compressed on‑disk; they are expanded only inside registers, keeping resident memory at the 14 MB blob size.

Performance on edge hardware

Device Prefill speed Decode speed
Raspberry Pi 5 800 tok/s 500 tok/s
Meta Quest 3S / Apple Vision Pro 400–1,500 tok/s (varying cores) —
$200‑class phones (Samsung A‑Series) 300–700 tok/s —
ESP32‑S3 (with external PSRAM) Runs within 28 MB RAM ceiling —

These numbers show that Needle 2 can operate fully offline on devices that lack GPUs or NPUs, delivering interactive latency for voice‑controlled assistants.

Tool‑calling accuracy versus larger models

Needle 2 is purpose‑built for agentic tool use and structured extraction. On five public benchmarks it matches or exceeds much larger models despite a 5×–70× smaller size.

Mobile Actions (961 rows)

  • Needle 2 accuracy: 63.7 % (name accuracy 98.3 %)
  • FunctionGemma 270M: 64.0 % overall, 87.3 % name accuracy
  • LFM2.5 230M: 69.1 % overall, 93.0 % name accuracy
  • Apple FM (on‑device): 57.6 % overall

DroidCall (200 rows)

  • Needle 2: 17.0 % overall, 36.5 % name accuracy
  • FunctionGemma 270M: 17.5 % overall, 37.5 % name accuracy
  • LFM2.5 230M: 11.0 % overall

Seal‑Tools (in‑domain, 700 rows)

  • Needle 2: 32.6 % overall, 64.9 % name accuracy
  • LFM2.5 230M: 26.9 % overall, 45.4 % name accuracy
  • FunctionGemma 270M: 16.3 % overall, 56.0 % name accuracy

Seal‑Tools (out‑of‑domain, 654 rows)

  • Needle 2: 28.7 % overall, 58.7 % name accuracy
  • LFM2.5 230M: 17.0 % overall
  • FunctionGemma 270M: 15.6 % overall

BFCL v4 single‑turn (3,641 rows)

  • Overall score: Needle 2 42.6 vs Apple FM 61.7, LFM2.5 60.8, FunctionGemma 46.1
  • Well‑formed rate: Needle 2 93.4 % (comparable to FunctionGemma’s 100 %)

Conclusion: When the task is exact tool invocation, Needle 2’s specialized training and grammar enforcement let it compete with models 5–6× larger.

Real‑world deployment example

Pebble’s Index 01 wearable runs Needle 2 locally to translate spoken commands into actions without any network dependency. The device has no screen, so reliability and latency are critical; the 14 MB model satisfies both constraints.

"The Pebble Index Ring has no screen. So when you speak to it, the action just has to happen, every time, with or without internet connection. We run Cactus Needle locally in the app, instead of relying on the cloud. The model's footprint is tiny and the performance never lets us down." – Pebble engineering team

Community feedback highlights

"The micro‑LLM space is underappreciated; a hierarchy of models where larger agents train tiny specialists could become a common pattern." – nater5000

"Confidence gating is crucial. In ambiguous queries the model sometimes returns a low‑confidence call instead of abstaining." – grenli

"Running on ESP32‑S3 is possible, but documentation for flashing the binary is still missing." – forsalebypwner

"The model’s reasoning field appears after the function call, which can be confusing for downstream pipelines." – alex7o

"A 2‑bit format was chosen over 4‑bit to keep the binary under 14 MB; 4‑bit would have increased size without proportional quality gain." – r0ze-at-hn

These comments underline both enthusiasm for the on‑device agentic paradigm and practical concerns about tooling, confidence calibration, and deployment ergonomics.

Limitations and open questions

  • General knowledge – Needle 2 is not a chat model; it lacks world knowledge beyond device actions, so open‑ended queries often return low‑confidence or incorrect calls.
  • Confidence calibration – The threshold for escalating to the cloud is learned but not publicly documented; users report occasional over‑confident incorrect calls.
  • Tool‑schema flexibility – Adding new functions requires fine‑tuning on a Mac/PC; the process is fast (minutes to hours) but still a barrier for non‑technical developers.
  • Hardware support – The binary currently targets Cortex‑M, x86, and WASM; attempts to run on generic 64‑bit ARM desktops have failed for some users.

Why Needle 2 matters for the edge AI ecosystem

Needle 2 demonstrates that agentic capability does not require billions of parameters when the problem is framed as structured function mapping. By co‑designing architecture, quantization, and runtime, Cactus Compute delivers a model that fits within the memory and compute envelope of the majority of IoT devices (≈80 % of edge hardware costs < $200). This shifts the AI deployment model from cloud‑centric APIs to private, instant, and offline assistants, opening new use‑cases in hearing aids, home automation, robotics, and low‑cost smartphones.

Getting started

  • Model weights – Apache 2.0‑licensed checkpoint on Hugging Face: Cactus-Compute/needle-2.
  • Runtime – Dependency‑free C++ binary (source on GitHub) automatically selects optimal kernels; also available as a WASM module for browsers.
  • Fine‑tuning – Python package needle lets you add custom tool definitions and train on a laptop in a few hours.
  • Demo – The web playground showcases voice‑to‑action scenarios for wearables, smart homes, and robots.

Needle 2 proves that a 14 MB LLM can reliably perform on‑device tool calling, offering a practical path to ubiquitous, privacy‑first AI across the billions of cheap edge devices that now exist.

Sources

Related