cactus-compute/needle

What it solves

Needle 2 is a highly compact, 45M-parameter model designed for tool calling, device control, and structured data extraction on tiny devices. It addresses the the need for AI agents that can run locally with minimal memory footprints (around 28MB RAM) without sacrificing the ability to perform complex tool-use tasks that typically require much larger models.

How it works

The project uses a "Simple Attention Network" architecture featuring a Hadamard MLP, GQA attention, and engram key-value memory. To achieve its extreme size, the model is compressed to 2-bit quantization (CQ2-bit) and baked into a single 14MB binary engine. It employs a byte-level grammar to constrain decoding, ensuring that tool calls are returned as valid JSON based on the provided schemas. It also includes a built-in retrieval head to manage large tool catalogues by selecting only the top five tools per turn.

Who it’s for

It is intended for developers building AI agents for resource-constrained environments, such as wearables, smart home devices, and other embedded systems where memory and power are limited.

Highlights

• Extreme Efficiency: A 14MB binary that runs in ~28MB of RAM, competing with models 5x to 70x larger.
• Confidence Gating: Every response includes a calibrated confidence score to allow for threshold-based escalation.
• Structured Output: Uses a compiled byte-level grammar to guarantee JSON outputs based on tool schemas.
• Integrated Pipeline: Includes tools for synthetic data generation, LoRA fine-tuning via JAX, and exporting to a compact .cact format.
• Pre-built Environments: Provides ready-made tool surfaces for domains like smart homes, media players, and wearables.