youssofal/MTPLX

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

What it solves

MTPLX provides a high-performance way to run large language models (LLMs) locally on Apple Silicon Macs. It specifically addresses the speed limitations of standard decoding by utilizing Multi-Token Prediction (MTP) heads to draft and verify multiple tokens at once, significantly increasing generation speed without sacrificing model accuracy.

How it works

MTPLX implements an exact speculative decoder using the model's own MTP heads. It drafts several tokens ahead of the current position and then uses a single batched forward pass to verify those drafts. Tokens are committed using exact rejection sampling with residual correction, ensuring the output remains identical to what the model would have produced via plain decoding. To optimize performance, it includes an "Auto-tune" feature that measures the specific hardware (chip, memory bandwidth, thermals) of a Mac to determine the ideal draft depth for a given model.

Who it’s for

It is designed for macOS users with Apple Silicon (M1 or newer) who want to run powerful models like Qwen 3.8, Gemma 4, or Ternary Bonsai 2 locally for coding, chatting, or powering AI agents via an OpenAI/Anthropic-compatible API.

Highlights

  • High-Speed Decoding: Achieves up to 2.24x speedups over plain decoding on high-end M-series chips.
  • MTP Integration: Native support for models with Multi-Token Prediction heads (e.g., Qwen 3.8 family).
  • Comprehensive Tooling: Includes a native Mac app with a live performance dashboard, a CLI, and a "Forge" tool to create and verify new MTP models from Hugging Face repos.
  • API Compatibility: Serves an OpenAI and Anthropic-compatible API, supporting tool calls, streaming, and integration with clients like Cursor, Cline, and Claude Code.
  • Efficient Memory Management: Supports various quantization levels (4-bit, 8-bit) and streams large n-gram tables from SSD to save RAM.

Related

  • Project
    Blaizzy/mlx-vlm

    A package for inference and and fine-tuning of Vision Language Models and Omni Models on Mac using MLX, supporting multimodal inputs and speculative decoding for faster generation.

  • Project
    jjang-ai/mlxstudio

    A native macOS app for Apple Silicon that provides a user-friendly interface to run LLMs, vision models, and image generation locally using the MLX framework.

  • Project
    ARahim3/mlx-dspark

    mlx‑d​spak is a Python package / macOS app that runs loss‑less speculative decoding (DSpark & DFlash) on Apple‑silicon Macs, giving 2‑5× faster text generation for many open‑source LLMs and exposing a local OpenAI‑compatible API.

  • Project
    mizorewww/laya-mlx

    A native Apple Silicon MLX port of Laya for high-speed, structured "typed decisions" (choices, scores, probabilities) without token-by-token text generation.

  • Project
    ml-explore/mlx-lm

    A Python package for generating text and fine-tuning large language models specifically optimized for Apple silicon using the MLX framework.