waybarrios/vllm-mlx

OpenAI and Anthropic compatible server for Apple Silicon. Run LLMs and vision-language models (Llama, Qwen-VL, LLaVA) with continuous batching, MCP tool calling, and multimodal support. Native MLX backend, 400+ tok/s. Works with Claude Code.

What it solves

It provides a high-throughput inference server specifically for Apple Silicon Macs, bringing advanced serving features like continuous batching and paged KV caching—typically found in enterprise servers—to local hardware. It eliminates the need for manual model conversion and allows users to run a variety of multimodal models (text, image, video, audio) through a single unified process.

How it works

Built on Apple's MLX framework and Metal kernels, the server leverages unified memory to run models natively on the GPU. It implements a vLLM-style architecture including a paged KV cache for memory efficiency, a trie-based prefix cache to speed up repeated prompts, and an SSD-tiered cache to handle long-context agents by spilling data to disk. It exposes these capabilities via OpenAI-compatible and Anthropic-compatible APIs.

Who it’s for

Developers and AI researchers using M-series Macs who need a production-grade local inference server capable of handling concurrent requests, multimodal inputs, or complex agentic workflows with long contexts.

Highlights

  • Dual API Support: Simultaneously exposes OpenAI /v1/* and Anthropic /v1/messages endpoints.
  • Advanced Memory Management: Features continuous batching, paged KV cache, and SSD-tiered caching for long contexts.
  • Multimodal Versatility: Supports LLMs, vision models (e.g., Qwen3-VL, Llama vision), audio STT (Whisper), and native TTS in one server.
  • Performance Optimizations: Includes speculative decoding, sparse prefill for faster time-to-first-token, and MoE expert reduction.
  • Developer Tooling: Built-in Prometheus metrics and a benchmarking tool for prompt sweeps.