cactus-compute/needle

Needle – A Tiny‑Device Foundation Model

What it is – Needle is a family of foundation models (the latest is Needle 3) that fit on very small devices – mobiles, wearables, robots, smart‑home hubs, automotive ECUs and even micro‑controllers. The whole model is shipped as a single 8‑29 MB binary (.cact file) and runs on a custom C++ engine that works on CPUs, WASI, browsers and other edge runtimes.

Core capabilities

Feature	What you get
Tool calling	Given a list of Python functions (tools) you expose, Needle selects the right ones and fills every argument from natural‑language input. It can return multiple calls in order or an empty list when no tool matches.
Structured extraction	Declare a JSON schema (shape). Feed messy text and Needle returns typed fields that always parse, with a calibrated confidence score.
Text embedding	The same model can emit a dense vector for any sentence, enabling local semantic search, routing or similarity matching.
Speech‑to‑text (Whistle)	A 16.9 MB speech model that shares Needle’s engine. One‑pass transcription (up to 30 s) with language detection, timestamps, keyword biasing, and streaming support.
Confidence scoring	Every response includes a learned confidence head, letting applications decide whether to act, ask for clarification, or reject.

How it works

• Laddered Simple Attention Network – a “Simple Attention” backbone where each depth (2‑20 layers) is a fully deployable sub‑model. The architecture uses a Monarch Hadamard MLP instead of a traditional feed‑forward network, GQA attention with causal convolution taps, an engram n‑gram memory, and multi‑lane hyper‑connections.
• Parameter efficiency – most weights live in the engram; a 121 M‑parameter model behaves like a 50 M one.
• Byte‑level grammar – a compiled grammar derived from your JSON schemas constrains token generation, guaranteeing syntactically valid output.
• Quantisation – weights are stored at 2.125 bits per weight (Cactus Quants) and the shipped binaries are 8‑29 MB.

Getting started

pip install cactus-needle          # core library + runtime
pip install "cactus-needle[train]"  # optional training utilities

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")['results'])
# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

• The run call returns a JSON object containing function_calls, the model’s internal reasoning trace, and a confidence score.
• For speech, simply call needle.transcribe("clip.wav") or use the streaming API needle.stream(chunks).

Benchmarks

• Tool calling – exact‑match accuracy on full test splits surpasses models 10× its size.
• Structured extraction – field‑level micro‑F1 beats larger baselines.
• Whistle – 8.6× smaller and 6.6× faster to first token than Whisper‑base, with lower word‑error rates on LibriSpeech, SPGISpeech, Earnings‑22, FLEURS, etc. (see the benchmark SVGs in the repo and the detailed tables on the Cactus Compute site).

Customization & fine‑tuning

• LoRA‑based local fine‑tuning (needle finetune) – train adapters on attention projections while keeping the base frozen; export a tuned .cact with needle build.
• Full‑model platform fine‑tuning (needle platform finetune) – runs on Cactus GPUs, trains every depth, optionally re‑calibrates the confidence head, and returns tuned binaries for any layer count.
• Fine‑tuned sub‑networks (as small as 2 layers) can outperform 29 M‑parameter models like DeepSeek V4 Flash on specific tool sets.

Deployment

• Pre‑built engine binaries are provided per target platform (macOS‑arm64, linux‑arm64, WASI, etc.).
• Build a deployment folder with needle build --platform <folder> [--layers N] – the command fetches the appropriate engine and places the .cact weights beside it.
• Run the binary directly: ./macos-arm64/needle --model needle3.cact --tools tools.json --serve.
• Telemetry is enabled by default; disable with NEEDLE_TELEMETRY=0 and DO_NOT_TRACK=1.

Resources

• Model weights & engines – Hugging Face repos Cactus-Compute/needle3 and Cactus-Compute/whistle.
• Docs & guides – design tools, confidence handling, structured extraction, fine‑tuning, supported devices, the .cact file format, and porting notes are all on the Cactus Compute blog (links listed in the README).
• Interactive demo – try Needle in the browser at https://cactuscompute.com/needle.

Citation If you use Needle in research or a product, cite the provided BibTeX entry.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.