TrevorS/voxtral-mini-realtime-rs

What it solves

This project provides a high-performance, native Rust implementation for running Mistral's Voxtral Mini 4B (ASR) and Voxtral 4B (TTS) models. It enables real-time speech-to-text and text-to-speech capabilities both as a local CLI tool and directly within a web browser using WebGPU.

How it works

It uses the Burn machine learning framework and custom WGSL compute shaders to execute model inference. The project supports two primary paths: a native BF16 path for high-precision tasks and a quantized Q4 GGUF path optimized for speed and memory efficiency. The browser implementation uses WebAssembly (WASM) and WebGPU to handle model loading and inference, overcoming browser-specific memory and allocation constraints through techniques like sharded model loading and asynchronous tensor processing.

Who it’s for

This is for developers and researchers who need to integrate real-time speech processing into applications without relying on cloud APIs. It is particularly useful for those building privacy-focused or offline-capable AI tools that need to run on consumer hardware or directly in a user's browser.

Highlights

• Runs 4B parameter models entirely client-side in the browser via WebGPU.
• Achieves real-time performance for both ASR and TTS using Q4 GGUF quantization.
• Includes a complete CLI for local transcription and speech synthesis.
• Features custom WGSL shaders for optimized fused dequantization and matrix multiplication.
• Supports 20 preset voices across 9 languages for text-to-speech.