jjang-ai/vmlx

vMLX - Use MLX models easily - JANGQ (GGUF for MLX) - Not dependant on mlx_vlm

What it solves

vMLX is a self-hosted inference server designed specifically for Apple Silicon (M1/M2/M3/M4) that allows users to run Large Language Models (LLMs), Vision Language Models (VLMs), and image generation models locally without needing third-party API keys.

How it works

It leverages the MLX framework to provide an OpenAI, Anthropic, and Ollama compatible HTTP API. The system uses a sophisticated 5-layer cache architecture (including memory-aware prefix and disk caches) and supports advanced inference techniques like continuous batching, speculative decoding, and prompt lookup decoding to optimize performance. It also supports distributed inference, allowing models to be split across multiple Macs via network connections (Thunderbolt, Ethernet, WiFi).

Who it’s for

Developers and AI researchers using Apple Silicon Macs who want a high-performance, local alternative to cloud APIs for text, image, and audio AI tasks.

Highlights

  • Broad Model Support: Runs any MLX model, including text LLMs, vision models, multimodal omni models, and MoE architectures.
  • Distributed Compute: Ability to run massive models across multiple Macs using pipeline parallelism.
  • L2 Disk Cache: Persists prompt caches to SSD so that follow-up messages are instant even after server restarts.
  • Multi-API Compatibility: Native support for OpenAI, Anthropic, and Ollama wire formats.
  • Integrated Desktop App: Includes MLX Studio for a GUI-based chat interface, model management, and developer tools.
  • Advanced Quantization: Supports JANG adaptive mixed-precision to maintain higher quality at lower bit-widths.

Related

  • Project
    waybarrios/vllm-mlx

    A vLLM-style inference server for Apple Silicon that provides continuous batching and paged KV cache for high-throughput local serving of multimodal models.

  • Project
    cubist38/mlx-openai-server

    An OpenAI-compatible API server for running local MLX models on Apple Silicon, supporting text, multimodal, image, and audio models.

  • Project
    jundot/omlx

    oMLX is a macOS-native LLM inference server that optimizes local model performance through tiered KV caching and continuous batching, managed via a menu bar app and web dashboard.

  • Project
    jjang-ai/mlxstudio

    A native macOS app for Apple Silicon that provides a user-friendly interface to run LLMs, vision models, and image generation locally using the MLX framework.

  • Project
    raullenchai/Rapid-MLX

    Rapid-MLX is an open-source LLM inference server for Apple Silicon Macs, built on Apple's MLX framework. It serves open-weight models (like Qwen) via OpenAI- and Anthropic-compatible APIs, tuned for fast, reliable tool calling by coding agents — with claims of up to 4× faster decoding than Apple's reference server, plus continuous batching, persistent prefix caching, a CLI, a desktop app, and image-generation support.