bstnxbt/dflash-mlx

Lossless DFlash speculative decoding for MLX on Apple Silicon

What it solves

dflash-mlx implements DFlash, a speculative decoding technique designed to accelerate LLM inference on Apple Silicon. It solves the bottleneck of autoregressive generation by using a small draft model to predict multiple tokens in parallel, which are then verified by a larger target model in a single pass, significantly increasing tokens-per-second throughput without losing output quality.

How it works

The project uses a block-diffusion draft model (typically around 1B parameters) to generate 16 tokens in one pass. A larger target model then verifies these tokens in a single forward pass. If the target model agrees with the draft, the tokens are accepted; if not, the draft is truncated at the first error and the target model provides the correct next token.

Key technical optimizations for Apple Silicon include:

  • Tape-replay rollback: Records an innovation tape to efficiently restore the state of GatedDeltaNet models during verification failures.
  • Custom Metal Kernels: Includes specialized verify_qmm int4 quantized matrix multiplication kernels for M-class GPUs to speed up the verification step.
  • Tiered Prefix Cache: A two-level (RAM and SSD) cache system that stores snapshots of the target model's state to skip prefill for revisited prompts.
  • Positional Sparse Prefill: Allows prefilling the target model on a selected subset of tokens to reduce computation.

Who it’s for

Developers and researchers using Apple Silicon (M-series chips) who want to run large language models (like Qwen and Gemma) with significantly higher generation speeds.

Highlights

  • Lossless Acceleration: Every emitted token is verified against the target model, ensuring the output is identical to the target model's greedy decoding.
  • High Speedups: Benchmarks show speedups ranging from 1.3x to 4.3x depending on the model and prompt length.
  • OpenAI-Compatible Server: Includes a server that can be integrated with tools like aider, Continue, and Open WebUI.
  • Architecture Support: Optimized for Qwen3.5, Qwen3.6, and Gemma4 models.
  • Integrated Diagnostics: Provides detailed metrics and structured artifacts for debugging andP performance analysis.

Related

  • Project
    z-lab/dflash

    A lightweight block diffusion model for speculative decoding that enables efficient, high-quality parallel drafting to speed up LLM generation.

  • Project
    ARahim3/mlx-dspark

    A high-performance inference engine for Apple Silicon that uses DSpark and DFlash speculative decoding to accelerate LLM generation speed losslessly.

  • Project
    youssofal/MTPLX

    A native Apple Silicon runtime that accelerates local LLMs by utilizing built-in multi-token prediction heads to increase decoding speed without changing output quality.

  • Dispatch
    vLLM speculative decoding on AMD GPUs: performance and methods

    vLLM adds speculative decoding to AMD Instinct MI300X/MI355X GPUs, letting a lightweight draft model propose multiple tokens that the target model verifies in a single pass, which can double or more the output-token throughput depending on the draft method, model family, and proposal length.

  • Dispatch
    vLLM Speculators: Parallel Drafting for Speculative Decoding

    vLLM and the Speculators project introduce open-source support for P-EAGLE, DFlash, and DSpark, moving beyond autoregressive drafting to generate candidate token blocks in parallel for faster LLM inference.