waybarrios/vllm-mlx

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

What it solves

vllm-mlx is a high-throughput inference server designed specifically for Apple Silicon Macs. It addresses the need for a vLLM-style serving experience on macOS, providing advanced memory management and batching techniques that are typically missing in basic local LLM runners, while supporting a wide range of multimodal models (text, image, video, audio) from a single process.

How it works

The project leverages the MLX framework and Metal kernels to utilize Apple's unified memory architecture. It implements several key optimization techniques inspired by vLLM:

  • Continuous Batching: Increases throughput by processing multiple requests concurrently.
  • Paged KV Cache: Efficiently manages memory and allows for prefix sharing.
  • Prefix Caching: Uses a trie-based system to share common prompt prefixes across requests.
  • SSD-Tiered Cache: Allows spilling the prefix cache to disk to support long-context agents.

It exposes compatible APIs for both OpenAI and Anthropic, allowing it to act as a drop-in replacement for those services in many local development environments.

Who it’s for

Developers and AI researchers using Apple Silicon (M1 through M5) who need to run multimodal models locally with high performance, high concurrency, and compatibility with existing AI SDKs and coding agents (like Claude Code or GitHub Copilot CLI).

Highlights

  • Dual API Support: Native support for OpenAI /v1/* and Anthropic /v1/messages endpoints.
  • Multimodal Capabilities: Single-server support for LLMs, vision models (e.g., Gemma 3, Qwen3-VL), audio (TTS and STT), and embeddings.
  • Advanced Memory Management: Features paged KV cache and SSD-tiered caching for long-context tasks.
  • Native Acceleration: Built on MLX for direct GPU acceleration on Apple Silicon without requiring model conversion steps.
  • Observability: Includes built-in Prometheus metrics and a benchmarking tool for prompt sweeps.

Related

  • Project
    jundot/omlx

    oMLX is a macOS-native LLM inference server that optimizes local model performance through tiered KV caching and continuous batching, managed via a menu bar app and web dashboard.

  • Project
    jjang-ai/vmlx

    A self-hosted inference server for Apple Silicon that provides OpenAI, Anthropic, and Ollama compatible APIs for running LLMs, VLMs, and image generation models locally.

  • Project
    SharpAI/SwiftLM

    A native Swift, OpenAI-compatible inference server for MLX models on Apple Silicon, featuring TurboQuant (4× KV-cache compression), MTP speculative decoding, and SSD streaming for large MoE models, with extensive benchmarks on M-series chips.

  • Project
    cubist38/mlx-openai-server

    An OpenAI-compatible API server for running local MLX models on Apple Silicon, supporting text, multimodal, image, and audio models.

  • Project
    vllm-project/vllm-metal

    vLLM Metal is a plugin that enables the vLLM LLM serving system to run on Apple Silicon Macs using the MLX/Metal backend. It provides Metal‑optimized attention kernels (paged varlen, M5 NAX prefill) and supports large models such as Qwen‑3.8‑27B. Install via Homebrew or a curl script, then use the regular `vllm` command on macOS 15+.