Blaizzy/mlx-audio-swift

A modular Swift SDK for audio processing with MLX on Apple Silicon

What it solves

MLX Audio Swift provides a unified, modular Swift SDK for integrating advanced audio AI capabilities into macOS and iOS applications. It eliminates the need for complex Python environments by allowing developers to run high-performance audio models—including text-to-speech, speech-to-text, and speaker diarization—directly on Apple Silicon using the MLX framework.

How it works

The SDK is organized into modular components that developers can import individually to keep application sizes small. It leverages the MLX framework for hardware acceleration on Apple Silicon and integrates with HuggingFace Hub for automatic model downloading. The system supports several specialized pipelines:

  • TTS (Text-to-Speech): Converts text to audio waveforms.
  • STT (Speech-to-Text): Transcribes audio into text.
  • VAD/Diarization: Detects voice activity and identifies different speakers in a recording.
  • Codecs: Encodes and decodes audio into tokens for efficient processing.
  • STS (Speech-to-Speech): Handles audio-to-audio transformations.

Who it’s for

Swift developers building audio-centric applications for macOS (14+) and iOS (17+) who want to implement on-device AI audio processing without relying on external APIs or Python.

Highlights

  • Modular Design: Import only the specific modules (e.g., MLXAudioTTS, MLXAudioSTT) required for your project.
  • Extensive Model Support: Compatible with a wide array of models including Whisper, Qwen3, and Fish Audio.
  • Apple Silicon Optimized: Built specifically for M-series chips using MLX Swift.
  • Sreaming Support: Capable of real-time audio generation for TTS.
  • SwiftUI Integration: Includes a dedicated MLXAudioUI module for building audio interfaces.

Related

  • Project
    soniqo/speech-swift

    A library of on-device AI speech models for Apple Silicon, providing local speech recognition, synthesis, and understanding for Mac and iOS apps.

  • Project
    FluidInference/FluidAudio

    A Swift SDK for local, low-latency audio AI on Apple devices, providing transcription, text-to-speech, and speaker diarization offloaded to the Apple Neural Engine.

  • Project
    Blaizzy/mlx-audio

    MLX‑Audio is a Python library (with CLI, web UI, and Swift package) that runs a large collection of modern speech and music models on Apple Silicon via the MLX framework. It supports TTS, STT, speech‑to‑speech, voice cloning, VAD/diarisation, and music generation, offering quantised 3‑ to 8‑bit variants for low‑memory inference. Install with `pip install mlx-audio`, generate audio via `mlx_audio.tts.generate …` or the Python `load_model(...).generate` API, and optionally serve an OpenAI‑compatible REST API.

  • Project
    argmaxinc/argmax-oss-swift

    A collection of on-device inference frameworks for Apple platforms providing speech-to-text, text-to-speech, and speaker diarization using Core ML.

  • Project
    MacPaw/OpenAI

    A community-maintained Swift SDK for the OpenAI API, enabling Apple developers to integrate text, image, and audio AI models into their applications.