FluidInference/FluidAudio

Frontier CoreML audio models in your apps — text-to-speech, speech-to-text, voice activity detection, and speaker diarization. In Swift, powered by SOTA open source.

What it solves

FluidAudio is a Swift SDK that enables developers to integrate high-performance, fully local audio AI capabilities into Apple devices (macOS and iOS). It eliminates the need for cloud-based audio processing, ensuring privacy and reducing latency by offloading inference to the Apple Neural Engine (ANE).

How it works

The SDK provides a set of optimized CoreML models that run directly on the ANE, avoiding the CPU and GPU to minimize power consumption and memory usage. It integrates open-source models for various audio tasks, which can be added to a project via Swift Package Manager or official wrappers for React Native and Rust/Tauri.

Who it’s for

It is designed for Apple platform developers building apps that require real-time or batch audio processing, such as dictation tools, meeting assistants, and voice-controlled interfaces.

Highlights

  • Automatic Speech Recognition (ASR): Supports batch and streaming transcription across multiple languages, including European languages, Japanese, and Mandarin Chinese.
  • Text-to-Speech (TTS): Features parallel synthesis with SSML and streaming TTS with voice cloning support.
  • Speaker Diarization: Provides both online (real-time) and offline (batch) pipelines to separate and identify different speakers in an audio stream.
  • Voice Activity Detection (VAD): Uses Silero models to detect when speech is occurring.
  • Apple Neural Engine Optimization: Specifically tuned for the ANE to ensure maximum performance and minimal power draw.
  • Inverse Text Normalization (ITN): Includes tools to convert spoken-form text (e.g., "two hundred") into written-form (e.g., "200").

Related

  • Project
    argmaxinc/argmax-oss-swift

    A collection of on-device inference frameworks for Apple platforms providing speech-to-text, text-to-speech, and speaker diarization using Core ML.

  • Project
    Blaizzy/mlx-audio-swift

    A modular Swift SDK for audio processing on Apple Silicon that provides easy integration of TTS, STT, and speaker diarization models using the MLX framework.

  • Project
    MacPaw/OpenAI

    A community-maintained Swift SDK for the OpenAI API, enabling developers to easily integrate text, image, and audio AI capabilities into Apple platform apps.

  • Project
    soniqo/speech-swift

    Speech Swift is an open‑source Swift package that provides dozens of on‑device speech AI models (ASR, TTS, translation, voice agents, enhancement, etc.) for Apple Silicon. It runs via Core ML and the MLX framework, supports hundreds of languages, offers multiple quantised sizes, and can be added to a project with a single Swift‑Package‑Manager dependency. The repo includes extensive docs, pre‑trained models, and example code for transcription, streaming dictation, and UI components, all without any cloud calls.

  • Project
    altic-dev/FluidVoice

    An open-source voice-to-text dictation app for macOS that enables on-device AI transcription and system control via voice commands.