ddalcu/mlx-serve

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Includes MLX Core macOS app with chat, agent mode, and tool calling.

What it solves

Running large language models and generative AI on macOS can be resource-intensive and complex. This project provides a high-performance, native way to run LLMs and multi-modal models (image, video, music, speech, and 3D) on Apple Silicon without the overhead of Python or Electron.

How it works

The project consists of a native Zig server that supports multiple model formats, including MLX and GGUF (via embedded llama.cpp). It exposes OpenAI, Anthropic, and Ollama-compatible APIs, allowing it to integrate seamlessly with existing tools like Cursor, Claude Code, and Open WebUI. It also includes a macOS menu-bar app called MLX Core for users who prefer a graphical interface for model management, chat, and agentic workflows.

Who it’s for

  • Developers who want a fast, local, and Python-free inference engine for their applications.
  • AI Enthusiasts looking for a high-performance way to run diverse models (text, image, video, etc.) on Mac hardware.
  • Power Users who want to use local models within their existing workflows via compatible APIs.

Highlights

  • Multi-modal support: Generates text, images, video, music, speech (with voice cloning), and 3D models.
  • High performance: Features include speculative decoding, continuous batching, and custom Metal kernels for fast prefilling.
  • Broad compatibility: Supports MLX and GGUF formats and provides OpenAI, Anthropic, and Ollama-compatible APIs.
  • Advanced features: Includes an agent sandbox (isolated Linux VM), LAN model sharing, and KV-cache quantization.
  • Zero-dependency runtime: Built in Zig, requiring no Python installation to run the binary.