Blaizzy/mlx-vlm

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

MLX‑VLM – Vision‑Language Model inference & fine‑tuning on Apple MLX

What it is – mlx‑vlm is a Python package that lets you run and fine‑tune modern Vision‑Language Models (VLMs) and multimodal “Omni” models (image + audio + video) locally on a Mac that supports Apple’s MLX framework. It bundles a CLI, a FastAPI server, and a Gradio chat UI, plus a collection of utilities for model conversion, quantisation, and speculative decoding.

Core capabilities

Capability	Details
Inference	`mlx_vlm.generate` supports text‑only, image, audio, or combined image‑+‑audio prompts.
Fine‑tuning	Tools (`mlx_vlm.convert`, `mlx_vlm.finetune` – not shown in the excerpt) to adapt Hugging‑Face checkpoints to the MLX format and apply low‑bit quantisation (4‑bit, 1‑bit affine, KV‑cache quant).
Speculative decoding	Draft‑model‑based speed‑ups using three families: DFlash/DFlash2/DSpark, Gemma‑4 MTP, EAGLE‑3. Users specify `--draft-model`, `--draft-kind`, and optional block‑size.
Thinking budget	For models that emit a “thinking” block (``), you can cap the tokens spent inside it (`--thinking-budget`).
Server / API	`mlx_vlm.server` launches a FastAPI service with optional continuous batching, automatic prefix caching, KV‑cache quantisation, and multi‑modal endpoints (LLM, image‑gen, TTS, STT).
Chat UI	One‑command Gradio interface (`mlx_vlm.chat_ui`) for interactive multimodal chat.
Model zoo	Ready‑to‑run wrappers for dozens of VLMs (e.g., Qwen2‑VL, Gemma‑4, MiniCPM‑o, Moondream 2/3, Granite Vision, LLaVA‑OneVision, etc.) with per‑model README links.
Agent‑skills bundle	A `skills/` directory containing ready‑made prompts for coding assistants (Claude Code, Codex, Gemini) to automate common tasks like conversion, benchmarking, or issue generation.

Quick start (CLI)

# Install the core package
pip install -U mlx-vlm
# Optional UI extra for Gradio
pip install -U 'mlx-vlm[ui]'

# Text generation
mlx_vlm.generate \
  --model mlx-community/Qwen2-VL-2B-Instruct-4bit \
  --prompt "Hello, how are you?" \
  --max-tokens 100

# Image generation
mlx_vlm.generate \
  --model mlx-community/Qwen2-VL-2B-Instruct-4bit \
  --image http://images.cocodataset.org/val2017/000000039769.jpg \
  --prompt "Describe this image." \
  --max-tokens 100

# Audio‑aware model
mlx_vlm.generate \
  --model mlx-community/gemma-3n-E2B-it-4bit \
  --audio /path/to/audio.wav \
  --prompt "What do you hear?" \
  --max-tokens 100

Speculative decoding example (speed‑up)

mlx_vlm.generate \
  --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --draft-kind dflash \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 \
  --temperature 0

The drafter proposes several tokens per step; the target model verifies them in a single forward pass, yielding up to ~4× faster generation for large models.

Python usage (minimal example)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")
prompt = "Describe this image."
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
formatted = apply_chat_template(processor, model.config, prompt, num_images=1)
output = generate(model, processor, formatted, image)
print(output)

The same API works for audio or combined image + audio inputs.

Server deployment

# Simple launch (defaults to port 8080)
mlx_vlm.server

# Pre‑load a specific model
mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit

# Enable thinking mode globally
mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

The server exposes endpoints for text generation, multimodal generation, and optional TTS/STT models.

Why it matters

Apple‑first – Leverages the MLX runtime, which runs efficiently on Apple Silicon GPUs without CUDA.
Broad multimodal support – Handles vision, audio, and video inputs, plus image‑generation outputs.
Performance tricks – Speculative decoding, KV‑cache quantisation, and 1‑bit affine inference give substantial speed/memory gains on‑device.
Developer friendliness – CLI, Python API, FastAPI server, and Gradio UI cover the whole workflow spectrum.

Getting help / contributing

The repository ships a skills bundle for coding‑assistant plugins (Claude Code, Codex, Gemini) that automates common development tasks.
Issues can be turned into reproducible GitHub reports via the reproducible-github-issues skill.
Contributions are guided by a contributing skill that checks pre‑commit hooks and test placement.

TL;DR – mlx‑vlm is a full‑stack toolkit for running and fine‑tuning vision‑language (and audio/video) models on Macs using Apple’s MLX, with a rich CLI, Python API, server, UI, and advanced speed‑up features like speculative decoding.