ddalcu/mlx-serve

What it solves

mlx-serve provides a high-performance, local inference server for Apple Silicon Macs, eliminating the need for Python or cloud dependencies. It allows users to run a vast array of LLMs (both MLX and GGUF formats) and multimodal models (image, video, music, speech, and 3D) on their own hardware with a unified, standard API surface.

How it works

Built as a native Zig binary, the server integrates several engines including MLX and an embedded llama.cpp for GGUF support. It exposes four compatible API surfaces on a single port: OpenAI, Anthropic, Ollama, and dedicated media endpoints. It utilizes custom Metal kernels, continuous batching, and four types of speculative decoding (PLD, draft companions, Gemma 4 drafter, and Qwen MTP) to maximize decode speeds on Apple Silicon.

Who it’s for

Developers and AI enthusiasts using Apple Silicon Macs who want a fast, standalone local inference server that integrates seamlessly with existing tools like Claude Code, Cursor, and Open WebUI without the overhead of Python or Electron apps.

Highlights

• Unified Model Support: Runs MLX and GGUF models, including massive architectures like DeepSeek V4 Flash.
• Multimodal Capabilities: Native generation of images, video, music, speech (with voice cloning), and 3D models.
• Multi-API Compatibility: Drop-in replacement for OpenAI, Anthropic, and Ollama APIs.
• High Performance: Up to 26% faster decode speeds than LM Studio on identical weights via advanced speculative decoding and Metal kernels.
• MLX Core App: A signed macOS menu-bar app providing a GUI for model management, chat, and an isolated Linux VM sandbox for agent shell commands.
• LAN Sharing: Ability to share models across multiple Macs on a local network via Bonjour.