vllm-project/vllm-metal

Community maintained hardware plugin for vLLM on Apple Silicon

vLLM Metal – Run vLLM on Apple Silicon GPUs

What it is

  • A plugin that lets the popular LLM serving system vLLM execute models on Apple Silicon Macs (M‑series chips) by using MLX as the compute backend and a Metal‑based attention kernel.
  • It bridges the gap between the PyTorch‑centric vLLM codebase and Apple’s GPU stack, providing a single lowering path for both frameworks.

Key components

  • Up‑stream vLLM – supplies the API server, request scheduler, and paged‑block memory manager.
  • mlx_lm – offers token‑wise model layers that run on MLX/Metal.
  • vllm‑metal – implements the request‑aware attention path, including:
    • A paged varlen kernel (default attention backend).
    • M5 NAX tensor units for fast multi‑head attention (MHA), GQA, and MQA prefill.
    • Support for speculative decoding.

Recent highlights

  • v0.2.0 (April 2026): unified paged varlen kernel, delivering ~83× faster first‑token latency and ~3.6× higher throughput vs. v0.1.0.
  • M5 NAX acceleration (August 2026): speeds up attention prefill.
  • Qwen‑3.8‑27B runs on Metal via the mlx‑community/Qwen3.8-27B-8bit model, fitting a 27‑billion‑parameter hybrid SDPA + GDN linear model on a single Mac.

Supported environment

  • macOS 15 (Sequoia) or later, Apple Silicon (M‑series). No support for Intel Macs or other OSes.

Supported models

  • A growing list of LLMs is documented in docs/supported_models.md. The README points to a matrix that shows which models have been verified to run on Metal.

Installation

  • Stable release (Homebrew)
    brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal
    After installation, the vllm command works directly—no virtual environment needed.
  • Development build (curl installer)
    curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate
    This fetches the latest code and creates an isolated Python environment.
  • Contributing – For developers who want to modify Python code or the Metal kernels, the repo provides a CONTRIBUTING.md with a development‑setup guide.

How to use

  1. Install as above.
  2. Run the standard vllm command (e.g., vllm serve --model <model-id>). The plugin automatically selects the Metal backend on supported Macs.
  3. Consult the Documentation link for detailed usage, configuration options, and performance‑tuning recipes based on available GPU memory.

Where to learn more

Bottom line vLLM Metal makes it practical to serve large language models locally on Apple Silicon, delivering substantial speedups through Metal‑optimized kernels while reusing the familiar vLLM API and scheduling logic.

Related

  • Dispatch
    vllm-metal v0.28.0 release notes / what's new

    vllm-metal brings vLLM's scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon, enabling efficient concurrent serving of LLMs via MLX and Metal.

  • Project
    waybarrios/vllm-mlx

    A vLLM-style inference server for Apple Silicon that provides continuous batching and paged KV cache for high-throughput local serving of multimodal models.

  • Dispatch
    Apple Silicon macOS VMs: Accelerating LLM Inference with Metal Capability Shim

    Cua has released a research shim that enables 11-16x faster LLM inference in macOS Virtualization.framework VMs by correcting Metal GPU capability reporting to unlock modern kernels.

  • Project
    mode-io/vllm-jev

    vLLM Jev is a fast HTTP serving wrapper for Jev‑style decision models (choice, yes/no, score). It runs on Linux GPUs via native vLLM and on Apple Silicon via MLX/PyTorch, handling text, images, and short videos. Install with `uv pip install .`, start a server with `vllm-jev serve <model-id>`, and query `/v1/systemone` for structured answers. Benchmarks show 4‑43× latency reductions versus reference implementations.

  • Project
    vllm-project/vllm

    vLLM is an open‑source library for high‑throughput, low‑latency serving of large language models. It introduces PagedAttention for efficient KV‑cache handling, supports many quantisation formats, integrates fast attention kernels, and offers an OpenAI‑compatible API that runs on GPUs, CPUs and other accelerators.