FluidInference/FluidAudio

Frontier CoreML audio models in your apps — text-to-speech, speech-to-text, voice activity detection, and speaker diarization. In Swift, powered by SOTA open source.

What it solves

FluidAudio is a Swift SDK that enables fully local, low-latency audio AI on Apple devices. It solves the problem of relying on cloud-based audio processing by offloading inference to the Apple Neural Engine (ANE), which reduces memory usage, lowers power consumption, and increases speed while maintaining user privacy.

How it works

The SDK integrates optimized open-source models (MIT/Apache 2.0) that run directly on the ANE, avoiding the CPU and GPU/MPS entirely. It provides a unified interface for several audio tasks:

  • Automatic Speech Recognition (ASR): Supports batch and streaming transcription across multiple languages using models like Parakeet TDT and SenseVoice.
  • Text-to-Speech (TTS): Parallel synthesis and streaming TTS with voice cloning via Kokoro and PocketTTS.
  • Speaker Diarization: Online and offline pipelines to separate and identify different speakers in an audio stream.
  • Voice Activity Detection (VAD): Uses Silero models to detect when speech is occurring.

Who it’s for

It is designed for macOS and iOS developers who want to build privacy-first, offline audio applications such as meeting assistants, dictation tools, and voice-controlled interfaces.

Highlights

  • Apple Neural Engine Optimization: Maximum performance and minimal power consumption by running inference on the ANE.
  • Comprehensive Audio Suite: Includes ASR, TTS, speaker diarization, and VAD in one SDK.
  • Multilingual Support: Transcription and synthesis across dozens of languages including English, European languages, Japanese, and Mandarin Chinese.
  • Local-First Architecture: Fully offline processing ensures data privacy and eliminates cloud latency.
  • Extensible Integration: Available as a Swift package with official wrappers for React Native/Expo and Rust/Tauri.