Cactus Needle 3 Release: 8‑29 MB Foundation Model for On‑Device Tool Calling and Structured Extraction

Needle 3 achieves DeepSeek‑level performance at a fraction of the size

Cactus Compute announced Needle 3, an 8‑29 MB binary foundation model that can match or beat DeepSeek V4 Flash on mobile tool‑calling and extraction benchmarks while running entirely offline on devices ranging from Raspberry Pi 5 to microcontrollers. The model’s small footprint makes it practical for smart‑home hubs, wearables, robots, and even embedded controllers.

---

Core capabilities delivered out‑of‑the‑box

• Tool calling – Needle 3 parses user utterances, selects the appropriate function from a developer‑provided toolbox, and fills every argument. It returns a JSON list of calls; if no tool matches, it returns an empty list instead of hallucinating.
• Structured extraction – By supplying a Pydantic schema, developers receive typed JSON objects from messy text (e.g., invoices, bookings). The decode grammar guarantees syntactic validity.
• Text embedding – The same model can emit sentence vectors for on‑device semantic search, duplicate detection, and routing.

These primitives enable end‑to‑end products such as:

• Voice‑controlled smart‑home commands ("Dim the bedroom and lock up").
• Robot instruction sequencing ("clean the kitchen but leave the bedroom").
• On‑device assistants for phones, wearables, AR glasses, and automotive cabins.

---

Intelligence laddering: a single binary, multiple sub‑networks

Needle 3 is built on Simple Attention Networks with CQ2‑bit quantisation. The 29‑layer backbone (29 L) contains 121 M parameters, but every prefix from 2 L to 20 L forms a functional sub‑network. Developers can pick the smallest sub‑network that satisfies latency or memory constraints, then fine‑tune it on a downstream task.

• 4‑layer (4 L) fine‑tuned model matches DeepSeek V4 Flash after a single epoch on task‑specific data.
• Sub‑networks range from 9 MB (2 L) to 29 MB (20 L), covering a wide spectrum of tiny hardware.
• Training used 360 B tokens of proprietary structured data, yielding 400‑4 000 tokens/s decode speed on a Raspberry Pi 5.

---

Deployment across 12 platforms with a sub‑MB engine

Cactus provides pre‑built inference engines (< 1 MB) for macOS, Linux (x86_64, arm64, armv7, riscv64, mipsel), Windows, Android, iOS, tvOS, watchOS, WebAssembly, and WASI components. The needle build --platform command fetches the appropriate engine and places the needle3.cact weight file beside it.

Platform	Engine artifact
macOS (arm64)	needle CLI, libneedle.a, needle.h
Linux (various)	same as macOS plus binaries for each ABI
Windows (x86_64/arm64)	needle.exe, libneedle.a, needle.h
Android / iOS / watchOS / tvOS	static library + header
Browser (Wasm)	needle.js, needle.wasm, needle.h
WASI component	needle.component.wasm, needle.wit

---

Developer workflow: from tool definition to execution

1. Define tools – Decorate Python functions; the signature supplies argument types, the docstring supplies the natural‑language description, and run() executes the call.
2. Optional triggers – Regular‑expression patterns force a call when matched, bypassing confidence gating.
3. Extraction – Pass a Pydantic model to extract() to obtain a typed object.
4. Confidence gating – Every response includes a calibrated confidence score (floor 0.1). Calls below the floor are omitted; higher scores can be used to decide whether to auto‑execute, ask for confirmation, or reject.
5. Fine‑tuning – Use the provided Python package for LoRA on the frozen base, then export a 4‑bit .cact file for any sub‑network.

---

Real‑world observations from the community

"turn all the lights on/off" and "it's too dark in the bathroom" worked, but less direct phrasing failed; thermostat commands sometimes inverted temperature direction. (gs17)

The model can handle up to ≈10 tools before confusion grows. (sourcecodeplz)

In a smart‑home demo, "warm the house" incorrectly adjusted lighting brightness instead of temperature, revealing bias toward lighting intents. (Retro_Dev)

Users reported occasional mis‑routing, e.g., "turn it up" lowered car volume, and the model sometimes produced unrelated actions like playing music for "I need a wee". (IanCal)

A participant noted that the model returns an empty list when no tool matches, which is preferable to hallucinated calls. (eriwang915)

Several commenters praised the on‑device privacy and the ability to run the model on constrained hardware such as ESP32‑based devices, though performance on newer ESP32‑P4 remains untested. (digdugdirk, viccis)

---

Limitations and open questions

• General reasoning: The model excels at tool selection and argument filling but still exhibits occasional nonsensical reasoning (e.g., mapping "cat puked" to vacuum actions). This reflects the trade‑off of sacrificing broad world knowledge for a tiny footprint.
• Dataset dependence: Community members emphasized that the quality of the underlying structured dataset is crucial for handling the myriad ways users phrase smart‑home commands.
• Hardware constraints: While inference runs on Raspberry Pi 5, it does not currently leverage Apple GPUs for inference, limiting performance on macOS devices.
• Comparison to larger open models: Benchmarks show Needle 3 outperforming Qwen 3.8 and approaching Gemma 4‑E4B on tool‑calling, but it still lags on generic classification tasks such as MNLI.

---

Why Needle 3 matters for on‑device AI

Needle 3 demonstrates that structured JSON output and reliable tool calling can be achieved with sub‑30 MB models, opening the door to privacy‑preserving assistants on devices that cannot host gigabyte‑scale LLMs. By providing a ladder of sub‑networks, Cactus lets developers balance latency, memory, and accuracy, making on‑device AI viable for smart‑home hubs, wearables, automotive infotainment, and industrial edge controllers.