Whistle 16.9 MB Speech‑to‑Text Model: Tiny, Fast, Multilingual On‑Device ASR

TL;DR – What Whistle delivers and why it matters

  • What: Whistle is a 16.9 MB on‑device speech‑to‑text model that runs entirely on the CPU, requires no external dependencies, and shares the same C++ runtime as the Needle language model.
  • Why it matters: At sub‑20 MB size it provides multilingual transcription, word‑level timestamps, and speech embeddings with latency‑to‑first‑token of ~11 ms—far faster than OpenAI’s Whisper‑base (73 ms) and comparable to much larger models, enabling real‑time STT on mobiles, wearables, micro‑controllers, and edge devices.

Core capabilities

  • Transcription – 16 kHz mono audio, up to 30 s, in English, German, French, Spanish, Italian, Dutch, Polish. Language is auto‑detected unless explicitly set.
  • Word timestamps – Each word is returned with start/end times and a confidence score, derived from the decoder’s attention mechanism.
  • Speech embedding – The encoder outputs a frame‑wise embedding (one 80 ms row per frame) without decoding, useful for downstream tasks such as speaker identification or audio retrieval.

"The first press downloads the 16.9 MB model, and audio never leaves your device." – Cactus Compute blog

Architecture at a glance

Component Design details Shared with Needle
Front‑end 25 ms window, 10 ms hop, 80 log‑mel bins, band‑limited 250‑3500 Hz, per‑channel normalisation. 30 s → 3,000 frames → 375 frames after three stride‑2 convolutions. Yes
Encoder 8 Simple Attention blocks (4 mHC residual lanes + Monarch Hadamard MLP). Non‑causal attention allows long‑range context (e.g., a frame at 3 s can attend to 12 s). Yes
Decoder 8 Laddered Simple Attention blocks, width 512, 8 query heads → 2 KV heads, 48‑dim Q/K, 64‑dim V, 3‑tap causal conv on Q/K/V, engram look‑ups (layers 3 & 7, 18,432 slots). Yes
Cross‑attention Gated cross‑attention per decoder layer: x ← x + σ(g)·softmax(q̂K̂ᵀ/√d)V. Encoder projections are computed once per clip (375 frames) and reused across beams. –
Decoding 5‑beam search, length‑normalised log‑probability, keyword biasing via Aho‑Corasick automaton, max 320 tokens, vocab 8,192 pieces + 7 language tokens. –
Silence handling Pre‑decode loudness check; below threshold returns empty transcript and language without entering beam search. –

Laddered depth selection

The --audio-depth flag selects a decoder depth at load time (2‑layer up to full 8‑layer). All encoder layers always run; the decoder can be sliced to trade speed for accuracy.

Benchmark highlights

All numbers are measured on an Apple M4 Pro CPU unless otherwise noted.

Metric Whistle Whisper‑base (145 MB) Moonshine tiny v2 (42 MB)
Model size 16.9 MB 145.3 MB 41.9 MB
Time‑to‑first token 11.1 ms (10 s clip) 73.2 ms 22.8 ms
Decode speed 1,319 tokens/s 266 tokens/s 262 tokens/s
LibriSpeech test‑clean WER ahead of Whisper & Moonshine – –
SPGISpeech WER ahead – –
Earnings‑22 WER ahead – –
FLEURS average WER ahead – –
TED‑LIUM WER behind Whisper – –
AMI WER behind Whisper – –
MLS average WER behind Whisper – –

"Whistle is ahead on LibriSpeech test‑clean and test‑other, on SPGISpeech, on Earnings‑22 and on the FLEURS average." – Cactus Compute blog

Interpretation of results

  • Speed: Whistle’s first‑token latency scales with clip length (5.9 ms at 5 s, 36.3 ms at 30 s), whereas Whisper pads all inputs to 30 s, giving a flat latency.
  • Accuracy: For English‑centric benchmarks Whistle matches or exceeds Whisper‑base, despite being >8× smaller. Multilingual performance is competitive on the seven supported languages, though some community comments note degradation on Spanish and Dutch‑German confusion.
  • Resource usage: The entire pipeline runs on a single CPU core with no GPU, making it suitable for low‑power devices (micro‑controllers, RISC‑V, MIPS, Android, iOS, etc.).

Integration and API surface

Command‑line usage (same binary for text and speech)

needle --model whistle.cact --audio clip.wav
needle --model needle3.cact --tools tools.json --prompt "turn off the kitchen lights"
needle --model needle3.cact --model whistle.cact --tools tools.json --audio clip.wav

The third command performs end‑to‑end voice‑controlled tool invocation, returning JSON:

{
  "function_calls": [{"name": "set_lights", "arguments": {"room": "kitchen", "on": false}}],
  "confidence": 0.94,
  "audio_text": "turn off the kitchen lights",
  "audio_language": "en"
}

Python wrapper (pip package cactus-needle)

import needle
print(needle.transcribe("clip.wav")["text"])  # → turn off the kitchen lights

Optional flags:

  • word_timestamps=True – returns per‑word timing and probability.
  • keywords=["Siobhan", "Krzysztof"] – boosts those phrases during beam search.
  • language="de" – forces German detection.
  • needle.Whistle() – creates an object for repeated embed(audio) calls.

Cross‑platform binaries

Pre‑built binaries exist for 17 targets, including macOS (ARM/x86), Linux, Android, iOS, watchOS, Windows‑ARM, RISC‑V, MIPS, browsers (WebAssembly), and a WASI component. Each folder contains:

  • needle executable
  • libneedle.a
  • needle.h All binaries load any .cact model file without environment‑variable configuration.

Community feedback (selected HN comments)

  • Positive reception: Users praise the tiny footprint and speed, especially for embedding in web apps or edge devices. (e.g., "For something you could plausibly ship inside a webapp, it’s very good.")
  • Accuracy concerns: Several commenters report language‑specific issues (Spanish, Dutch vs. German, Polish) and occasional hallucinations on challenging audio. One user observed a 60‑second segment repeatedly outputting "Thank you." indicating a failure mode on long dialogues.
  • Feature requests: Real‑time streaming transcription (output while recording) is missing; users note it as essential for live applications.
  • Comparisons: Users compare Whistle to Parakeet, Whisper, and Moonshine, noting Whistle’s speed advantage but mixed accuracy relative to larger models.
  • Hardware curiosity: Questions about running on micro‑controllers (ESP32, STM) and RISC‑V were raised; the project already ships a RISC‑V build, but full micro‑controller support remains unverified.

Practical use‑cases

  1. Voice‑controlled IoT devices – Local transcription eliminates latency and privacy concerns for smart‑home commands.
  2. Robotics – On‑board speech interfaces for robots without GPU acceleration.
  3. Wearables & automotive – Low‑power, on‑device STT for hands‑free interaction.
  4. Edge inference pipelines – Combine Whistle embeddings with a small LLM for context‑aware command parsing.

How to get started

  1. Download the model and runtime
    needle download whistle   # pulls the 16.9 MB .cact file
needle download macos-arm64   # example for Apple Silicon
  2. Run a quick test
    ./macos-arm64/needle --model whistle.cact --audio clip.wav --audio-word-timestamps
  3. Integrate via Python – install cactus-needle and call needle.transcribe or needle.embed as shown above.
  4. Explore depth trade‑offs – experiment with --audio-depth 2 … 8 to balance speed vs. WER for your target hardware.

Limitations and open questions

  • Language coverage: Only seven European languages are supported; no Chinese, Japanese, or other high‑demand languages.
  • Streaming output: Current API provides a final transcript after the audio clip is fully processed; real‑time token‑by‑token streaming is not yet available.
  • Robustness to accents: Community reports indicate degraded performance on strong accents (e.g., Polish, Dutch‑German, Indian English), suggesting the training data may be biased toward standard accents.
  • Micro‑controller feasibility: While a RISC‑V binary exists, running on ultra‑low‑power MCUs (ESP32, STM32) has not been demonstrated publicly.

Where to find the code and models

Whistle demonstrates that high‑quality, multilingual speech‑to‑text can be delivered in a sub‑20 MB CPU‑only package, opening the door to truly offline voice interfaces on a wide range of edge hardware.

Sources

Related

  • Project
    cactus-compute/needle

    Needle is a 8‑29 MB foundation model for edge devices that can do tool‑calling, structured JSON extraction, text embedding and speech‑to‑text (via the Whistle model). It uses a laddered Simple Attention architecture where every depth is a runnable sub‑model, and it ships as a single `.cact` binary with a custom C++ runtime. The Python SDK lets you decorate functions as tools, run the model locally, fine‑tune with LoRA or full‑model training, and deploy to many platforms.

  • Dispatch
    Needle 2 14 MB Agentic LLM Brings On‑Device Tool Calling to Sub‑$200 Devices

    Needle 2 is a 45 M‑parameter, 14 MB LLM that runs entirely on cheap edge hardware, delivering 500 tokens/sec decoding and reliable tool‑calling while using only 28 MB of RAM.

  • Project
    openai/whisper

    Whisper is OpenAI’s open‑source, transformer‑based speech‑to‑text system that handles transcription, language identification, and translation across many languages. It ships as a pip‑installable package with six model sizes (tiny → large, plus a fast “turbo” variant). The CLI (`whisper …`) and a simple Python API (`whisper.load_model(...).transcribe(...)`) let you turn audio files into text with just a few commands. The project runs on Python 3.8‑3.11, PyTorch, and requires ffmpeg (and optionally Rust for the tokenizer). All code and model weights are MIT‑licensed.

  • Project
    SYSTRAN/faster-whisper

    faster-whisper is a Python library that re‑implements OpenAI’s Whisper speech‑to‑text model using the high‑performance CTranslate2 inference engine. It delivers up to 4× faster transcription with similar accuracy, supports CPU, GPU and 8‑bit quantisation, and offers features like word‑level timestamps, VAD filtering and batched inference. Install via pip, pick a model size (e.g., "large‑v3"), create a WhisperModel with the desired device/precision, and call `transcribe`. Ideal for live captioning, batch processing, or any project needing Whisper‑quality transcription with lower latency and memory use.

  • Dispatch
    Cactus Needle 3 Release: 8‑29 MB Foundation Model for On‑Device Tool Calling and Structured Extraction

    Cactus Compute’s Needle 3 delivers an 8‑29 MB foundation model that matches or exceeds DeepSeek V4 Flash on tool‑calling and extraction tasks, enabling offline, structured JSON responses on tiny devices.