raullenchai/Rapid-MLX

What is it?

Rapid-MLX is a local AI inference server purpose-built for Apple Silicon Macs (M1–M4). It loads open-weight LLMs like Qwen and runs them entirely on your own machine, exposing standardized OpenAI- and Anthropic-compatible APIs so coding agents (Codex CLI, Claude Code, Aider, etc.) can talk to it as a drop-in replacement for cloud APIs — no code leaves your Mac.

It's built on Apple's MLX machine-learning framework, and its main selling point is speed: the README claims up to 4× faster decoding than Apple's reference mlx-lm server (1.5× on a typical task) with identical models and weights. Benchmark tables in the README show it outpacing alternatives like oMLX and LM Studio on Qwen models.

Key capabilities

• Dual API compatibility — speaks both the OpenAI chat API (/v1/chat/completions, /v1/responses) and the Anthropic Messages API, so most existing client tools work without adapters.
• Tool calling for agents — 27 parser modules for extracting structured tool/function calls from model output, with an auto-detect fallback. This is the core use case: reliable function calling for coding agents.
• Performance engineering — continuous batching for concurrent requests, a radix prefix cache (optimizes repeated prompt prefixes across requests) that persists to disk across restarts, and quantized KV cache.
• Multiple runtimes — CLI, a macOS desktop app (GUI for chat, model management, vision, files), and an HTTP server.
• Beyond text — includes image generation (e.g., FLUX, SDXL, Stable Diffusion 3.5), image editing, and vision models; audio (TTS/STT) and embeddings are opt-in extras.
• Model catalog — curated alaises (e.g., qwen3.5-4b-4bit) with download sizes, RAM requirements, and recommended defaults. First-run rapid-mlx chat auto-downloads a starter model (~3 GB).

How you'd use it

# Install
brew install rapid-mlx

# Chat directly
rapid-mlx chat

# Or serve for other apps (OpenAI-compatible endpoint)
rapid-mlx serve qwen3.5-4b-4bit
# → http://localhost:8000/v1

# Or launch a coding agent pre-wired to use it
rapid-mlx launch claude-code

Any client that accepts a custom OpenAI/Anthropic endpoint can point at localhost:8000 and start using local models.

What to watch for

• Hardware lock-in — requires an M-series Mac (Apple Silicon); no Windows/Linux builds. Memory matters: small 4-bit models run in ~8–12 GB unified RAM; larger ones need 32–96 GB.
• Performance claims are vendor-measured — the README's benchmarks are single-run comparisons on specific Macs (e.g., Mac Studio M5 Max, Mac mini M4 Pro), so real-world speed varies with hardware, power state, and concurrency.
• Actively developed — Apache 2.0 license, CI, frequent releases, community benchmark program.

Verdict

A substantial, well-engineered local-inference project tackling a real pain point: running LLMs for coding agents on Apple silicon with competitive speed. The dual API compatibility and tool-calling focus make it unusually plug-and-play with existing agent tooling.