Blaizzy/mlx-audio-swift

A modular Swift SDK for audio processing with MLX on Apple Silicon

What it solves

MLX Audio Swift provides a modular Swift SDK for integrating high-performance audio AI models into macOS and iOS applications. It eliminates the need for complex setup by offering a unified, type-safe API to access a wide variety of audio processing tasks on Apple Silicon.

How it works

The SDK is built on the MLX framework and is organized into modular components that developers can import individually to keep application sizes small. It supports automatic model downloading from HuggingFace Hub and utilizes native Swift async/await for asynchronous processing. The library is split into several specialized modules:

  • MLXAudioCore: Base utilities and types.
  • MLXAudioTTS: Text-to-Speech generation.
  • MLXAudioSTT: Speech-to-Text transcription.
  • MLXAudioVAD: Voice Activity Detection and speaker diarization.
  • MLXAudioSTS: Speech-to-Speech processing.
  • MLXAudioCodecs: Audio codec implementations for encoding and decoding audio tokens.
  • MLXAudioUI: SwiftUI components for building audio interfaces.

Who it’s for

Swift developers building for macOS (14+) and iOS (17+) who want to integrate AI-powered audio features like transcription, voice synthesis, and speaker identification on Apple Silicon hardware.

Highlights

  • Modular Design: Import only the specific modules (e.g., TTS or STT) required for your app.
  • Extensive Model Support: Includes implementations for a vast array of models including Whisper, Qwen3, and Soprano.
  • Apple Silicon Optimized: Specifically tuned for M-series chips using the MLX framework.
  • Real-time Capabilities: Supports streaming audio generation for low-latency TTS.
  • Automatic Integration: Simplifies model loading from HuggingFace with a single line of code.

Related

  • Project
    Blaizzy/mlx-audio

    An audio processing library optimized for Apple Silicon that provides fast inference for text-to-speech, speech-to-text, speech-to-speech, and music generation.

  • Project
    soniqo/speech-swift

    Speech Swift is an open‑source Swift package that provides on‑device speech AI for Apple Silicon. It bundles many modern ASR, TTS, translation, and voice‑agent models (e.g., Qwen3‑ASR, Whisper‑ASR, Omnilingual ASR, Qwen3‑TTS, CosyVoice, PersonaPlex) built on MLX and Core ML, so you can run inference locally with no cloud calls. Each model is a separate Swift‑Package‑Manager product, making it easy to add only what you need. The repo includes quick‑start code, SwiftUI UI helpers, Homebrew install, and extensive documentation.

  • Project
    ml-explore/mlx-swift-lm

    A Swift package for building applications with LLMs and VLMs using MLX Swift, featuring model loading, fine-tuning, and grammar-constrained generation.

  • Project
    ml-explore/mlx-swift-examples

    A collection of example programs and libraries for MLX Swift, demonstrating how to build LLM, VLM, and image generation apps on iOS and macOS.

  • Project
    PowerBeef/Vocello

    A private, local-first voice studio for macOS and iOS that uses MLX and Swift to generate high-quality speech via built-in voices, voice design, and voice cloning.