waybarrios/vllm-mlx

vllm‑mlx – High‑throughput LLM serving on Apple Silicon

What it is

• A drop‑in, vLLM‑style inference server that runs large language models (and multimodal models) directly on Apple Silicon GPUs using the MLX framework and Metal kernels.
• It bundles a number of performance tricks from the original vLLM project—continuous batching, paged KV‑cache, prefix sharing, and an optional SSD‑backed cache—so many concurrent requests can be handled with low latency.
• The server presents both OpenAI‑compatible (/v1/chat/completions, /v1/completions, /v1/embeddings, etc.) and Anthropic‑compatible (/v1/messages) REST APIs, letting existing client libraries talk to it unchanged.

Why it matters

• Apple’s M‑series chips have powerful GPU cores, but most open‑source LLM servers target CUDA. vllm‑mlx lets developers exploit the native Metal backend without converting models to another format.
• Continuous batching and the paged KV cache dramatically improve throughput and memory usage, especially for long‑context or multi‑turn chat scenarios.
• Built‑in support for vision, audio, and embeddings means a single process can serve text‑only LLMs, vision‑language models, speech‑to‑text, text‑to‑speech, and vector‑search embeddings.

Key features (as described in the README)

Category	Highlights
APIs	OpenAI‑compatible endpoints (/v1/*), Anthropic /v1/messages, tool‑calling parsers for 19 different model families, JSON‑Schema‑based structured output.
Performance	Continuous batching, paged KV cache, prefix cache, optional SSD‑tiered KV cache, warm‑prompt preloading (1.3‑2.25× faster first‑token), MoE top‑k reduction, speculative decoding, sparse prefill.
Multimodal	Vision models (Gemma 3/4, Qwen3‑VL, Pixtral, Llama‑vision), image/video/audio inputs, native TTS (11 voices, 15+ languages) and STT (Whisper family, up to 197× real‑time on M4 Max).
Advanced reasoning	Reasoning extraction parsers, MoE expert reduction, speculative decoding, attention‑based prefill.
Observability	Prometheus metrics endpoint, built‑in benchmark CLI (vllm‑mlx bench‑serve).
Hardware	Apple Silicon only (M1‑M5) via MLX/Metal, unified memory, no model conversion step.

Typical workflow

1. Install – pip install vllm-mlx (or uv tool install vllm-mlx). Optional audio extras with pip install vllm-mlx[audio].
2. Start the server – e.g. vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching.
3. Call it – Use the OpenAI Python SDK or Anthropic SDK pointing at http://localhost:8000/v1 (no API key needed). The README shows a minimal chat example and a curl example for reranking.
4. Optional extras – Warm‑prompt loading, SSD cache directory, MoE top‑k, speculative decoding, tool‑calling, multimodal payloads, TTS/STT utilities, embedding models, etc.

Performance numbers (M4 Max, 128 GB)

• LLM decode speed: up to ~418 tokens/s for a 0.6 B 8‑bit model, ~206 t/s for a 3 B 4‑bit model, ~128 t/s for a 30 B 4‑bit MoE model.
• Speech‑to‑text: Whisper‑tiny runs at 197× real‑time, Whisper‑large‑v3‑turbo at 55×, Whisper‑large‑v3 at 24×.

Use cases

• Local development of LLM‑powered apps on MacBooks without needing a cloud GPU.
• Prototyping multimodal agents (text + image + audio) that require a single unified API.
• Running private, offline inference for sensitive data on Apple hardware.
• Benchmarking and research on continuous‑batching and KV‑cache strategies on non‑CUDA GPUs.

Installation & getting started

# Recommended uv install (system‑wide CLI)
uv tool install vllm-mlx
# Or pip in a virtualenv
pip install vllm-mlx
# Audio extras
pip install vllm-mlx[audio]
brew install espeak-ng   # needed for non‑English TTS

Full docs are hosted at https://vllm-mlx.is-a.dev/ and include step‑by‑step guides for server configuration, multimodal usage, benchmarking, and model acquisition.

License & community

• Apache 2.0 open‑source license.
• Actively maintained by Wayner Barrios and the MLX community; contributions welcome (bug fixes, performance work, new benchmarks, etc.).

---

In short, vllm‑mlx brings the high‑throughput serving capabilities of vLLM to Apple Silicon, exposing familiar OpenAI/Anthropic APIs while supporting text, vision, audio, and embeddings—all without leaving the Mac ecosystem.