Cactus Needle: A 26M Parameter Function Calling Model

Overview

Cactus Needle is an open-source function calling model with only 26 million parameters, designed to outperform significantly larger models in single-shot tool invocation. It is optimized for edge deployment, capable of reaching prefill speeds of 6,000 tokens per second and generation speeds of 1,200 tokens per second on consumer hardware.

Architecture: The Simple Attention Network

Needle utilizes a "simple attention network" that differs fundamentally from the standard Transformer architecture by removing all Multi-Layer Perceptrons (MLPs) and dense layers.

Removal of Knowledge Stores

In a typical Transformer, the feed-forward network (FFN) layers are believed to act as the model's knowledge store, where facts and world knowledge are retained. Cactus Needle removes these layers entirely, operating solely on attention and gating. This design choice is based on the premise that tool calling is fundamentally a task of retrieval and assembly—matching a query to a tool name and extracting arguments—rather than a task requiring deep reasoning or extensive internal knowledge.

Technical Specifications

• Structure: A 12-encoder stack utilizing self-attention with Group Query Attention (GQA).
• Design: An encoder-decoder architecture rather than a decoder-only GPT style.
• Training: The model was pre-trained on 200 billion tokens using 16 TPUs for 27 hours, followed by post-training on 2 billion tokens of synthetic function calling data across 15 tool categories.
• Optimization: The team employed quantization-aware training to ensure the model remains performant when compressed for edge devices.

Performance and Benchmarks

Cactus Needle is designed for single-shot function calling. In this specific domain, it reportedly outperforms models that are 10x to 20x larger, including Function Gemma, IBM Granite, LFM, and Qwen 600M.

However, the model is not intended to replace large language models (LLMs) for complex tasks. It is likely to struggle with:

• Parallel function calling.
• Tasks requiring complex reasoning over multiple tools.
• "Skills" involving progressive disclosure.

Practical Application and Deployment

Because of its tiny footprint, Needle can be deployed on low-power hardware such as mobile devices, Raspberry Pis, or embedded systems (e.g., smart appliances).

Zero-Shot and Fine-Tuning Capabilities

Needle demonstrates strong zero-shot capabilities when provided with a JSON definition of tools. Even with functions it was not explicitly trained on—such as smart home controls generated by Claude—the model can accurately map user intent to the correct function and extract arguments.

For specialized use cases, the model can be fine-tuned on a CPU without requiring a GPU. The provided GitHub repository includes JAX code (using Flax and Optax) to facilitate this process.

The Cascade Architecture Strategy

Cactus advocates for a "cascade architecture" where small, specialized models handle specific tool sets. In this system, a small model can act as a router, deciding which tool set a query belongs to and then delegating the task to another specialized small model or falling back to a larger cloud-based model for complex reasoning.