Blaizzy/mlx-vlm

What it is

MLX-VLM is a Python package that lets you run (inference) and fine-tune Vision Language Models (VLMs) — AI models that understand images, videos, and audio — directly on your Mac. It's built on Apple's MLX machine-learning framework, which is optimized for Apple Silicon (M1/M2/M3 chips).

The README describes a feature-rich tool: a command‑line interface, a web chat UI (Gradio), a local server (FastAPI) with continuous batching and prefix caching, speculative decoding for speed, multi-image chat, fine‑tuning, and support for dozens of specific model architectures (Qwen, Gemma, MiniCPM, Phi, DeepSeek‑OCR, and many more).

What makes it interesting

• Runs heavy multimodal models locally on a Mac — no cloud needed, no GPU server.
• Inference & fine‑tuning in one package, with conversion utilities to turn Hugging Face models into MLX format.
• Speculative decoding acceleration: uses a small “drafter” model to guess the next tokens, then verifies them with the big model, often 2–3× faster. Supports several drafter families (DFlash, EAGLE‑3, MTP).
• Omni models — models that handle not just images but also audio and video, including speech generation.
• Production‑ready server with continuous batching (serving multiple requests efficiently), automatic prefix caching, and KV‑cache quantization to save memory.
• Extensive model‑specific documentation — each supported architecture has its own README with prompt formats and best practices.

How it works

• Install: pip install -U mlx-vlm (the UI needs mlx-vlm[ui]).
• Use via CLI: e.g., mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-4bit --image http://... --prompt "Describe this image".
• Programmatic: load a model and processor, then call generate(); supports images, audio, and text.
• Server: mlx_vlm.server --model ... exposes a REST API (FastAPI) with endpoints for generation, chat, audio, caching, and metrics.
• Quantization: convert Hugging Face models to MLX with mlx_vlm.convert, choosing bit‑widths (e.g., 4‑bit) to reduce memory.
• Fine‑tuning: the package includes fine‑tuning capabilities (mentioned in the table of contents, though details are not in this README excerpt).

Notable details

• Thinking budget: for reasoning models, you can cap how many tokens they spend thinking before answering.
• Multi‑image chat: supported for several models, allowing you to pass multiple images in a conversation.
• Distributed inference and 1‑bit affine inference are also covered.
• The project ships agent skills — ready‑made prompts for coding assistants (Claude Code, Codex, Gemini) that help them follow the project’s conventions.

Bottom line

MLX‑VLM is a practical, well‑documented toolkit for anyone who wants to run and customize modern vision‑language and omni models on a Mac, with real‑world features like speculative decoding, a local server, and fine‑tuning. It’s a genuine software project, actively maintained, with a large matrix of supported models.