soniqo/speech-swift

Speech Swift – On‑device speech AI for Apple Silicon

What it is

• A Swift package that bundles a large collection of modern speech‑related models (ASR, TTS, alignment, translation, voice‑agents, enhancement, etc.) and runs them locally on Macs and iOS devices.
• It uses Apple’s Core ML runtime (including the Neural Engine) and the open‑source MLX framework to execute the models without any cloud service or API key.

Why it matters

• All processing stays on the device, so you keep privacy, avoid latency, and can work offline.
• The library supports a huge range of languages (up to 1 672 for the Omnilingual ASR) and many tasks, from simple dictation to full‑duplex speech‑to‑speech agents.
• Models are provided in several quantised forms (FP16, INT8, INT5, INT4, etc.) so you can trade off speed, memory and quality to fit the hardware you target.

Key capabilities

Category	Example models (size)	What they do
Speech‑to‑Text (ASR)	Qwen3‑ASR (0.6‑1.7 B), Whisper‑ASR (Large‑v3 Turbo), Omnilingual (300 M‑7 B, 1 672 languages)	Convert audio to text, with optional speaker diarisation and timestamps.
Alignment	Qwen3‑ForcedAligner (0.6 B)	Produce word‑level timestamps for a given transcript.
Text‑to‑Speech (TTS)	Qwen3‑TTS (0.6‑1.7 B), CosyVoice (0.5 B), Kokoro (82 M)	Generate natural‑sounding speech, many with voice‑cloning and emotion tags.
LLM & Translation	Qwen3Chat (0.8‑4 B), FunctionGemma (270 M), MADLAD‑400 (3 B)	Run a language model locally; translate between 400+ languages.
Speech‑to‑Speech / Voice Agents	PersonaPlex (7 B), VoiceChat (11 B)	Full‑duplex audio‑in/audio‑out agents that can also run tool‑calling LLMs.
Enhancement & Separation	DeepFilterNet3 (2.1 M), Source‑Separation (HTDemucs/UMX), FlashSR (audio super‑resolution)	Denoise, remove echo, separate music stems, up‑sample audio.
Turn detection & Diarisation	Wake‑word, VAD, Smart‑Turn, Speaker Diarisation	Detect when a user starts/stops speaking and who is speaking.

How to add it

// In your Package.swift
.package(url: "https://github.com/soniqo/speech-swift", branch: "main")

Then import only the pieces you need, e.g.:

import ParakeetStreamingASR   // streaming ASR
import SpeechUI               // optional SwiftUI helpers

Quick code snippet – transcribe a buffer

import ParakeetStreamingASR

let model = try await ParakeetStreamingASRModel.fromPretrained()
let text  = try model.transcribeAudio(audioSamples, sampleRate: 16_000)
print(text)

The same model can be used for live streaming:

for await part in model.transcribeStream(audio: samples, sampleRate: 16_000) {
    print(part.isFinal ? "FINAL: \(part.text)" : "… \(part.text)")
}

A tiny SwiftUI view (SpeechUI) lets you drop a dictation UI into an app in ~10 lines.

Ecosystem & community

• Documentation – full guides at https://soniqo.audio (model‑specific pages, benchmark tables, deployment tips).
• Pre‑trained models – hosted on Hugging Face under the aufklarer org.
• Blog & Discord – regular posts about performance tricks, new model releases, and a Discord server for support.
• Homebrew – install the command‑line tools via brew install speech.
• Verified downstream projects – 16 public repos (e.g., AnythingLLM, Voicey, DexDictate) already depend on Speech Swift.

Typical use‑cases

• Voice assistants that run entirely on‑device (no network calls).
• Transcription apps for journalists, podcasters, or accessibility tools.
• Audio generation such as podcast narration, audiobooks, or in‑game NPC speech.
• Real‑time communication with low‑latency speech‑to‑speech translation.
• Audio post‑processing – noise suppression, echo cancellation, source separation.

Performance Benchmarks (iPhone 16 Pro, Core ML) show real‑time factors as low as 0.04 RTF for streaming dictation and 0.08 RTF for high‑quality TTS, meaning the model processes audio faster than it is recorded.

License Most models are released under permissive licences (Apache‑2.0, MIT) or non‑commercial research licences; the Swift package itself is open‑source under the MIT licence.

---

Bottom line – Speech Swift is a comprehensive, production‑ready toolbox that brings the latest speech‑AI capabilities to Apple devices without leaving the device, making it ideal for privacy‑first apps that need high‑quality speech recognition, synthesis, translation, or voice‑agent functionality.