mizorewww/laya-coreml

Local Laya typed decisions on Apple Core ML and Neural Engine. Validated ports, ~5 ms short decisions on M3 Max, reproducible speed and energy benchmarks.

What it solves

Laya-CoreML provides a way to run decision-making AI models locally on Apple Silicon devices with high efficiency. Unlike standard generative AI models that produce text token-by-token, this tool performs a single forward pass to output calibrated probabilities for specific choices, scores, or true/false questions. It eliminates the need for heavy frameworks like PyTorch or Transformers, resulting in faster performance and lower energy consumption.

How it works

It converts Laya model weights into Core ML programs that run natively on the CPU, GPU, or Apple's Neural Engine (ANE). When you provide a prompt and a set of constraints (such as a list of options or a rubric), the model analyzes the input and returns a structured decision result. Because it is not a generative model, it does not produce text tokens or require JSON parsing, making it ideal for automated decision-making workflows.

Who it’s for

This project is for developers building local, high-performance AI agents on macOS who need fast, reliable decision-making capabilities. It is particularly useful for applications requiring low-latency responses, such as game logic, routing, or classification tasks where energy efficiency and local execution are priorities.

Highlights

  • Zero token generation: Returns direct probabilities for choices, scores, or propositions.
  • Optimized for Apple Silicon: Leverages the Neural Engine for sub-5ms decision times in specific configurations.
  • Energy efficient: Uses significantly less power than compiled MLX alternatives.
  • Flexible integration: Supports CLI, HTTP, and MCP server modes, with integrations for LangChain, LlamaIndex, and others.
  • Offline-capable: Allows for fully local operation by pinning model bundles and forbidding network access.

Related

  • Dispatch
  • Project
  • Project
  • Dispatch
  • Project