jjang-ai/vmlx

What it solves

vMLX is a self-hosted inference server designed specifically for Apple Silicon (M1/M2/M3/M4). It allows users to run large language models (LLMs), vision-language models (VLMs), and image generation models locally without needing third-party API keys, providing an OpenAI, Anthropic, and Ollama compatible HTTP API.

How it works

The project leverages the MLX framework to optimize performance on Mac hardware. It implements a sophisticated 5-layer cache architecture (including memory-aware prefix and disk caches) and supports advanced techniques like continuous batching, speculative decoding, and paged KV caching to increase speed and memory efficiency. For models too large for a single machine, it supports distributed inference via pipeline parallelism across multiple Macs.

Who it’s for

It is built for developers and AI enthusiasts using Apple Silicon who want to host their own models locally, integrate them into applications via standard APIs, or run massive models across a cluster of Macs.

Highlights

• Broad Model Support: Runs text LLMs, vision models, multimodal omni models, MoE, hybrid SSMs, and image generation/editing models (Flux).
• Cros-API Compatibility: Supports OpenAI, Anthropic, and Ollama wire formats.
• Distributed Compute: Ability to split transformer layers across multiple Macs via Thunderbolt, Ethernet, or WiFi.
• Advanced Quantization: Supports standard MLX quantization and JANG adaptive mixed-precision for higher quality at lower bit-widths.
• Integrated Desktop App: Includes MLX Studio for model management, chat UI, and developer tools like GGUF-to-MLX conversion.
• Multimodal Capabilities: Integrated support for text-to-speech (Kokoro) and speech-to-text (Whisper).