A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.

MLX‑Audio – Fast, Apple‑Silicon‑native audio AI library

What it is – A Python package (and optional Swift package) that wraps a large collection of state‑of‑the‑art speech and music models and runs them on Apple’s MLX framework. It provides ready‑to‑use command‑line tools, a Python API, a web UI, and an OpenAI‑compatible REST endpoint for:

Text‑to‑Speech (TTS)
Speech‑to‑Text (STT / ASR)
Speech‑to‑Speech / source‑separation
Voice Activity Detection & speaker diarization
Music generation
Quantized inference (3‑ to 8‑bit) for low‑memory Apple Silicon devices

Quick start (CLI)

# install the package
pip install mlx-audio   # or: uv tool install mlx-audio

# generate a short TTS clip (default voice "Vivian")
mlx_audio.tts.generate \
    --model mlx-community/Qwen3-TTS-12Hz-0.6B-CustomVoice-8bit \
    --text "Hello, world!" \
    --play

Add --stream to hear audio while it is being generated, --save to write to disk, or --join_audio to merge multi‑segment output into a single file.

Quick start (Python)

from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Qwen3-TTS-12Hz-0.6B-CustomVoice-8bit")
for result in model.generate(
        "Hello from MLX‑Audio!",
        voice="Vivian",
        lang_code="English",
    ):
    # result.audio is an mx.array containing the waveform
    print("generated", result.audio.shape[0], "samples")

The same pattern works for STT, VAD, diarization, etc., using the corresponding mlx_audio.stt or mlx_audio.vad modules.

Core features (from the README)

Apple‑Silicon optimisation – inference runs on the MLX backend, delivering low‑latency generation on M‑series chips.
Broad model catalog – dozens of pre‑hosted models on Hugging Face (e.g., Kokoro, OmniVoice, Qwen3‑TTS, Whisper, VAD, music‑generation models). Most have 8‑bit, 4‑bit, or 3‑bit quantized variants.
Multilingual & voice‑cloning – many models support 20‑plus languages; several (OmniVoice, Higgs Audio, CSM) can clone a speaker from a reference audio clip.
Streaming & real‑time control – --stream flag for TTS, streaming ASR models (e.g., Nemotron 3.5‑ASR‑streaming, VibeVoice‑ASR), and full‑duplex VoiceChat.
Web UI – an interactive browser interface with 3‑D audio visualisation.
OpenAI‑compatible REST API – drop‑in replacement for services like OpenAI’s audio/speech endpoint.
Quantization – support for 3‑, 4‑, 6‑, 8‑bit (and MXFP formats) to fit large models into the limited memory of laptops.
Swift package – optional iOS/macOS integration for native apps.

Model categories (excerpt)

Category	Example models	Languages / notes
TTS	Kokoro, OmniVoice, Qwen3‑TTS, Higgs Audio v3, Voxtral‑TTS	20‑+ languages, voice cloning, speed control
STT / ASR	Whisper‑large‑v3‑turbo, Qwen3‑ASR, VibeVoice‑ASR, Moonshine	99+ languages (Whisper), streaming variants
VAD / Diarization	Silero VAD, Sortformer v1/v2.1	language‑agnostic, up to 4 speakers
STS / Enhancement	SAM‑Audio (source separation), DeepFilterNet, NemotronLabs VoiceChat	noise removal, full‑duplex speech chat
Music generation	MiniMax Music 3 (hierarchical AR + flow‑matching)	multilingual lyrics, 44.1 kHz stereo

Installation options

Method	Command	When to use
Standard pip	`pip install mlx-audio`	Simple local use, includes core library.
uv (CLI tools only)	`uv tool install --force mlx-audio`	If you only need the `mlx_audio.*` command‑line utilities.
From source (dev / server)	```bash
git clone https://github.com/Blaizzy/mlx-audio.git
cd mlx-audio
pip install -e ".[dev, server]"


---
## Running the web UI / API server
```bash
# after installing with the "server" extra
mlx_audio.server --host 0.0.0.0 --port 8000

The UI is reachable at http://localhost:8000 and offers a text box, voice selector, and a 3‑D waveform visualiser.

License & citation

License: MIT (per the badge in the README).
Citation: The README includes a Citation section; users are encouraged to cite the original model repositories (e.g., Qwen3‑TTS, Whisper) when publishing results.

Who might use this?

Developers building voice assistants or multimodal agents that need low‑latency speech synthesis or recognition on Mac laptops.
Researchers prototyping new audio models who want a unified interface and easy quantisation on Apple hardware.
iOS/macOS app developers looking for a Swift‑compatible audio‑AI SDK.
Content creators who want a local, offline TTS/voice‑cloning pipeline without cloud costs.

Bottom line

MLX‑Audio is a comprehensive, Apple‑Silicon‑focused toolbox that bundles dozens of cutting‑edge speech and music models, offers both CLI and Python APIs, supports aggressive quantisation, and even ships a web UI and OpenAI‑compatible server. It turns the otherwise heavyweight world of large‑scale audio models into something you can run locally on a MacBook.

Blaizzy/mlx-audio