raullenchai/Rapid-MLX

Rapid‑MLX – Fast, local AI on Apple Silicon

What it is – Rapid‑MLX is a Python‑based inference engine that runs large language, vision, audio and diffusion models natively on M‑series Macs using the MLX library. It ships a command‑line tool (rapid‑mlx) and a macOS desktop app, both of which expose an OpenAI‑compatible HTTP API (/v1/...) so any client that talks to OpenAI or Anthropic (e.g., LangChain, Aider, Claude Code, Codex CLI) can be pointed at http://localhost:8000 and run entirely offline.

---

Quick start (60 s)

# install (Homebrew, pip or the guided installer)
brew install rapid-mlx   # or curl … | bash

# chat in the terminal (downloads a 4‑bit 4‑B model on first run)
rapid-mlx chat

# or start an OpenAI‑compatible server
rapid-mlx serve qwen3.5-4b-4bit

You can now call the server with curl or any OpenAI SDK:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":"Say hello"}]}'

---

Core capabilities

Category	How it works	Example models
Text generation	Continuous‑batching kernels, prompt cache, quantised KV cache (int4/int8)	qwen3.5-4b-4bit, qwen3.6-35b-8bit
Vision / image generation	Diffusion pipelines (Flux, Stable Diffusion, etc.) accessed via /v1/images/*	flux2-klein-4b, z-image-turbo
Video generation	Text‑to‑video / image‑to‑video back‑ends (Wan, CogVideoX‑Fun, LTX)	wan2.2-ti2v-5b-q8
Audio	TTS, transcription, voice cloning, forced‑alignment via /v1/audio/*	kokoro, whisper-large-v3-turbo, indextts
Embeddings	Standard OpenAI /v1/embeddings endpoint	(same text models)
Agent integration	Pre‑verified adapters for 12 popular coding/assistant CLIs (Claude Code, Codex CLI, Aider, etc.)	–

---

Installation options

Method	What you get
Homebrew (brew install rapid-mlx)	Pre‑built binary, adds the rapid-mlx CLI to your PATH
Guided installer (`curl …	bash`)
pip/uv (pip install rapid-mlx)	Pure‑Python install; you can inspect or modify the package
Desktop app (download from rapidmlx.com)	One‑click GUI that bundles the same engine plus model manager

---

Using it as a drop‑in OpenAI/Anthropic backend

• Endpoint – http://localhost:8000/v1 (OpenAI) or http://localhost:8000 (Anthropic /v1/messages).
• Authentication – No API key is required for local use; you can set RAPID_MLX_API_KEY if you expose the server publicly.
• Supported routes – chat/completions, responses (Codex), messages (Claude), embeddings, images/generations, images/edits, videos, audio/*.
• Client compatibility – Works with LangChain, Pydantic‑AI, OpenCode, Aider, Claude Code, and any other tool that can be configured to talk to a local OpenAI‑compatible endpoint.

---

Agent support

Rapid‑MLX ships wire‑verified adapters for 12 coding agents. Five are marked Tier‑1 (Claude Code, Codex CLI, Hermes, Aider, DeepSeek Harness) and are re‑tested against real model weights on every release. The remaining agents (OpenCode, Qwen Code, OpenHands, Kilo Code, GitHub Copilot, Factory Droid, Moonshot Kimi Code) are Tier‑2 and also verified, though with a slightly lighter test suite.

You can auto‑configure an agent with a single command, e.g.:

rapid-mlx launch claude-code   # patches Claude Code to use the local server

---

Performance highlights (as claimed in the README)

• Up to 3× the throughput of Ollama on the same hardware (benchmark link provided).
• Image generation with flux2-klein-4b takes ~9 s per 1024×1024 image on an M3 Ultra.
• Video generation runs one clip at a time; a 1‑second clip with the Wan 2.2 model takes several minutes of compute.
• Quantised KV cache and continuous batching keep memory usage low enough to run 35 B models on Macs with 16 GB‑32 GB unified memory.

---

Limitations / gotchas

• Apple‑silicon only – The engine relies on MLX kernels that run on M1‑M4; no Windows or Linux binaries are provided.
• Single‑flight generation – Diffusion pipelines (image, video) are serialized; you cannot run two generations concurrently because of unified‑memory constraints.
• Model licensing – Rapid‑MLX does not re‑license the underlying models; you must respect each model’s original terms before commercial use.
• Public exposure – If you tunnel the server to the internet you must set RAPID_MLX_API_KEY; otherwise the endpoint is unauthenticated and should stay local.
• Python version – Core text/audio works on Python 3.10+, but the video runtime requires Python 3.11+.

---

Where to go next

• Docs – https://rapidmlx.com/docs/ (full API reference, model catalog, benchmark methodology)
• Model mirror – https://models.rapidmlx.com/ (download pre‑cached weights)
• Desktop app – https://rapidmlx.com/desktop for a GUI experience
• Community – Discord server (link in README) for help, model recommendations, and release announcements.

---

Bottom line – Rapid‑MLX is a production‑grade, locally‑run inference stack for Apple‑silicon Macs that mimics the OpenAI/Anthropic APIs, making it easy to replace cloud calls with fast, private inference for chat, coding assistants, image/video generation, and audio tasks.