Blaizzy/mlx-vlm

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

What it solves

MLX-VLM provides a streamlined way to run and fine-tune Vision Language Models (VLMs) and Omni Models (which support audio and video) specifically optimized for Apple Silicon Macs using the MLX framework. It simplifies the process of deploying these complex multimodal models locally with high performance.

How it works

The package leverages the MLX framework to enable efficient inference and fine-tuning. It provides a variety of interfaces for interaction, including a Command Line Interface (CLI), a Python API, a Gradio-based chat UI, and a FastAPI server. To increase generation speed, it implements speculative decoding using various drafter families like DFlash, EAGLE-3, and Multi-Token Prediction (MTP).

Who it’s for

Developers and AI researchers using Mac hardware who want to deploy, test, or fine-tune multimodal models that can process text, images, audio, and video.

Highlights

  • Multimodal Support: Handles text, images, audio, and video inputs across a wide range of supported models (e.g., Qwen, Gemma, MiniCPM).
  • Spped-up Inference: Supports speculative decoding (DFlash, EAGLE-3, MTP) to significantly increase token throughput.
  • Production-Ready Server: Includes a FastAPI server with continuous batching, automatic prefix caching, and KV cache quantization.
  • Thinking Mode: Support for "thinking" models (like Qwen3.5) with configurable token budgets for the reasoning blocks.
  • Local Optimization: Specifically designed for MLX to maximize the performance of Apple Silicon.