AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms

Overview

AnySpeechRecognizer and AnyDecisionModel are new Swift packages designed to provide developers with a consistent, modular API for integrating speech recognition and decision-making capabilities into Apple platform applications. These tools allow developers to swap between various local and hosted model backends—such as NVIDIA's Parakeet, Alibaba's Qwen3-ASR, and Cloudflare's Clef—while maintaining a unified codebase.

AnySpeechRecognizer

AnySpeechRecognizer extends Apple's native speech-to-text framework by allowing developers to plug in alternative engines while maintaining the familiar SpeechAnalyzer API. It supports local execution via Core ML and includes specialized modules for transcription and speaker diarization.

Supported Engines and Models

  • Parakeet Series: Includes Parakeet v3 (polyglot), Parakeet Ultra (optimized by Moondream), and Parakeet Unified (English-only). These models support live input and provide word-level timing and confidence scores.
  • Qwen3-ASR: Available in 0.6B and 1.7B parameter sizes, this model supports 30 languages and 22 Chinese dialects. It is optimized for file-based transcription rather than live streaming.
  • Speaker Diarization: The package introduces a SpeakerDiarizer module using NVIDIA's Streaming Sortformer, which can track up to four speakers simultaneously, filling a gap in Apple's native Speech framework.

Technical Implementation

Models are converted to Core ML and hosted on the Hugging Face Hub, with the package pinning specific revisions to ensure consistency. The package includes a command-line tool, hear, which allows developers to test transcription accuracy and evaluate performance on their own hardware. Performance benchmarks on an M4 Max Mac Studio show that Parakeet models can achieve word error rates (WER) as low as 1.43% on clean test data.

AnyDecisionModel

AnyDecisionModel provides a structured, type-safe approach to decision-making tasks, moving away from the traditional method of parsing free-form text generated by language models. It is designed to return probabilities for discrete choices, yes-or-no questions, and rating scales.

Capabilities and Backends

  • Probabilistic Output: Instead of text generation, the model returns typed results with associated probabilities and confidence scores, making it suitable for programmatic logic.
  • Backend Flexibility: The package supports multiple backends including MLX (for local Apple silicon execution), Core AI, llama.cpp, Jev, and Ollama.
  • Decision Heads: For advanced use cases, it supports models with dedicated decision heads, such as Cloudflare's Clef-Flash, which can process complex schemas in a single forward pass.
  • Debiasing: To mitigate common issues where models favor specific options based on list position, the package includes rotationDebiasing to average probabilities across multiple permutations of options.

Example Application: Emoji Magnet

To demonstrate the utility of AnyDecisionModel, the developers created "Emoji Magnet," a macOS application that dynamically filters emoji based on user-provided natural language descriptions. The app uses the model to evaluate hundreds of emoji candidates in real-time, showcasing how local inference can handle complex classification tasks without relying on external keyword rules.

Requirements and Getting Started

Both packages require Swift 6.2 or later and macOS 26 or iOS 26 or later. Local execution of Parakeet, Sortformer, and MLX-based decision models requires Apple silicon. Developers can integrate these tools by adding the dependencies to their Package.swift file, with specific traits available to enable local inference backends like MLX or Llama.

Sources