Cactus Needle 3 releases 8‑29 MB foundation model that matches DeepSeek V4 Flash on tool calls

Needle 3 delivers 8‑29 MB on‑device AI that matches DeepSeek V4 Flash on tool calls

Takeaway: Cactus Compute released Needle 3, a single 8‑29 MB binary that runs on tiny devices (microcontrollers, wearables, cars) and, when fine‑tuned on a single epoch, matches the performance of the much larger DeepSeek V4 Flash on tool‑calling and extraction tasks.

What Needle 3 is and why it matters

Needle 3 is a foundation model built on Cactus’s Simple Attention Network (SAN) architecture and quantised to 2‑bit CQ2. The model is released as a ladder of sub‑networks ranging from 2 to 20 layers, each a self‑contained model with monotonically increasing capacity. Developers can pick the smallest sub‑network that fits their hardware and fine‑tune it for a specific tool set. The 4‑layer (4L) sub‑network, after a single epoch of downstream fine‑tuning, reaches or exceeds DeepSeek V4 Flash on mobile‑tool‑call benchmarks despite being 10× smaller in parameter count.

Core capabilities

Tool‑calling

  • Needle reads a list of tool definitions (function signatures and docstrings) and selects the appropriate tool(s) for a user request.
  • It can emit multiple calls in a single turn, e.g., “Dim the bedroom and lock up” → [{"tool":"set_lights","args":…},{"tool":"lock_door","args":…}].
  • When no tool matches, Needle returns an empty list rather than hallucinating a call, preserving safety.

Structured extraction

  • Users declare a Pydantic schema; Needle returns a typed JSON object that conforms to the schema.
  • The decode grammar guarantees syntactically valid output, making the extraction reliable for invoices, bookings, notifications, etc.

Text embedding

  • The same model can produce a dense vector for a sentence, enabling on‑device semantic search, duplicate detection, and routing to the most relevant tool.

Performance numbers

Metric Needle 3 (4L, fine‑tuned) DeepSeek V4 Flash
Tool‑call accuracy (mobile) Matches or exceeds Baseline
Extraction accuracy 2‑3× larger models Comparable
Model size 8‑29 MB (CQ2‑quantised) ~300 MB
Inference speed on Raspberry Pi 5 400‑4k tokens/s decode, 1‑10k tokens/s pre‑fill —

The laddered architecture yields binaries from 9 MB (2L) up to 29 MB (20L), all fitting within a 1 MB engine wrapper for each target platform.

Deployment across platforms

Needle ships a pre‑built inference engine (<1 MB) and the needle3.cact weight file. The needle build --platform <target> command downloads the appropriate engine and places the weights beside it. Supported targets include:

Target Platforms
macOS macos-arm64
Linux x86_64, arm64, armv7, riscv64, mipsel
Windows x86_64, arm64
Android arm64, armv7, riscv64
iOS / tvOS / watchOS arm64 (device & simulator)
Browser WebAssembly (needle.js, needle.wasm)
WASI component wasm-component

All platforms load the same binary weights, enabling a single model to run on everything from a Raspberry Pi 5 to an ESP‑32‑class microcontroller (previous Needle releases have run on ESP32; community members are testing the new ladder on ESP32‑P4).

How to use Needle in an application

  1. Install the Python package – the inference engine is fetched from Hugging Face on first use and cached locally.
  2. Define tools – decorate Python functions; the signature provides argument types, the docstring supplies the tool description, and run() executes the call.
  3. Add optional triggers – regular‑expression patterns force a call when matched, bypassing the confidence floor.
  4. Call the model – needle.call(prompt, tools=…) returns a JSON object containing:
    • function_calls – the selected tool calls (or empty list).
    • confidence – a calibrated score (floor 0.1). Calls below the floor are suppressed.
    • results – the final response with tool results attached.
  5. Fine‑tune if needed – use LoRA on the frozen base, then export a 4‑bit .cact file for any sub‑network.

Community feedback highlights

"turn all the lights on/off" and "it's too dark in the bathroom" worked, but indirect phrasing often failed, and confidence scores were low for bad responses. – gs17

"I told the car to close the garage door and turn the car on and it didn't immediately refuse." – monster_truck

"from my limited tests, it can work with up to 10 tools/definitions; more than that confuses the model." – sourcecodeplz

"Tool‑call + structured JSON at 8–29 MB is the right bet. Returning an empty list when nothing fits beats inventing a fake tool." – eriwang915

"The intelligence laddering and on‑device tool calling are especially cool. Nice work getting this running across so many platforms!" – Natashash23

These comments illustrate both the promise (compact, safe tool calling) and current limits (sensitivity to phrasing, tool‑set size). The community is already experimenting with smart‑home, automotive, and robotics use cases, and many see potential for privacy‑preserving on‑device assistants.

When Needle 3 is the right choice

  • Privacy‑first applications – all inference runs locally; no data leaves the device.
  • Resource‑constrained hardware – runs on Raspberry Pi 5, ARM‑based phones, wearables, and microcontrollers.
  • Tool‑driven workflows – any scenario where the user request maps to a known function (smart‑home control, robot commands, on‑device search).
  • Structured data extraction – converting free‑form text into typed JSON without hallucination.

If you need open‑ended conversational abilities, large‑scale world knowledge, or multi‑turn reasoning beyond the defined tool set, a full‑size LLM remains more appropriate.

Future directions

Cactus plans to extend the ladder with more fine‑tuning recipes, improve the confidence calibration, and broaden support for edge‑accelerators (Apple GPU, specialized ASICs). Community contributions around tool‑definition best practices and dataset curation are encouraged, as the quality of the underlying structured dataset heavily influences real‑world performance.

Sources

Related