vllm-project/vllm-metal

Community maintained hardware plugin for vLLM on Apple Silicon

What it solves

It enables high-performance Large Language Model (LLM) inference on Apple Silicon Macs, allowing users to leverage the hardware's capabilities via the vLLM framework.

How it works

The project acts as a plugin for vLLM that uses MLX as the primary compute backend. It unifies MLX and PyTorch under a single lowering path to optimize execution on Mac hardware.

Who it’s for

Developers and researchers using Apple Silicon Macs who want to run vLLM for efficient LLM serving and inference.

Highlights

  • High-performance inference using MLX as the backend.
  • Support for M5 NAX tensor units to accelerate prefill for MHA, GQA, and MQA.
  • Unified paged varlen Metal kernel for improved Time To First Token (TTFT) and throughput.
  • Support for a growing list of models, including Qwen3.8.

相關

  • 專案
    lmstudio-ai/mlx-engine

    為 LM Studio 開發的基於 Apple MLX 的 LLM 引擎，可在 macOS 上實現文本與視覺模型的優化本地推理。

  • 專案
    vllm-project/tpu-inference

    A hardware plugin for vLLM that provides a unified backend for high-performance LLM inference on Google TPUs, supporting both PyTorch and JAX.

  • 專案
    jjang-ai/vmlx

    一個自託管的 Apple Silicon 推理伺服器，提供相容 OpenAI、Anthropic 與 Ollama 的 API，讓本機執行 LLM、VLM 與影像生成模型。

  • 專案
    ml-explore/mlx-lm

    一個用於在 Apple silicon 上使用 MLX 框架生成文字並微調大型語言模型的 Python 套件。

  • 專案
    baidu/vLLM-Kunlun

    A hardware plugin for vLLM that enables high-performance LLM inference on Kunlun XPU hardware with support for a wide range of models and quantization methods.