mizorewww/laya-mlx

Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.

What it solves

Traditional AI models often generate long text responses, which is slow and requires parsing to extract a simple answer. Laya-MLX solves this by directly outputting calibrated probabilities for predefined options, rubric levels, or true/false judgments in a single forward pass. This makes decisions like routing a support ticket, scoring a response, or checking a statement extremely fast (milliseconds) and avoids the overhead of token generation and JSON parsing. It runs entirely locally on Apple Silicon, so no cloud API is needed and your data stays private.

How it works

You define a set of questions using a schema. Each question has a type: choice (pick one from named options), score (rate against ordered levels), or noul (answer yes/no). You also provide instructions and, for choices, descriptions of each option. Laya-MLX then runs the Laya model (converted to MLX format) on your input, and returns probabilities for each possible answer, along with confidence scores. The model does not generate any text—it just computes probabilities from the encoder and decision heads. The package includes several pre-converted checkpoints (for English, multilingual, and typed-decisions workflows) that can be loaded with a single line of code. It also offers features like batching for shared question schemas, language routing, confidence gating, and tools for serving via CLI, HTTP, or MCP.

Who it's for

Developers building applications that need fast, local, and reliable decisions—for example, customer support triage, automated moderation, agent routing, or any system that must classify or judge without the latency of a full language model generation. It's especially suited for those working on Apple Silicon hardware who want private, offline inference. The library also integrates with common agent frameworks (LangChain, LlamaIndex, CrewAI, Agno), making it easy to drop into existing workflows.

Highlights

  • Runs natively on Apple Silicon (M3 Max benchmarks: 13.4 ms for a short English decision, 7.4 ms for multilingual)
  • No PyTorch, Transformers, or cloud API required—fully local inference
  • Supports choice, scoring, and yes/no questions with calibrated probabilities
  • Multiple checkpoints: English, multilingual, and typed-decisions variants
  • Batching and long-document scanning for high throughput (up to 395 q/s on M3 Max)
  • Confidence gating to flag low-confidence answers for review
  • Includes CLI, HTTP server, MCP server, and evaluation/calibration tools
  • Integrates with LangChain, LlamaIndex, CrewAI, and Agno
  • Apache-2.0 licensed, with detailed benchmarks and provenance documentation

Related

  • Dispatch
  • Dispatch
  • Project
  • Project
  • Project