jjang-ai/vmlx

vMLX - Use MLX models easily - JANGQ (GGUF for MLX) - Not dependant on mlx_vlm

What it solves

vMLX provides a self-hosted inference server specifically optimized for Apple Silicon (M1/M2/M3/M4), allowing users to run Large Language Models (LLMs), Vision Language Models (VLMs), and image generation models locally without needing third-party API keys.

How it works

It leverages the MLX framework to provide an OpenAI, Anthropic, and Ollama compatible HTTP API. The system uses a sophisticated 5-layer cache architecture (including memory-aware prefix, paged KV, and disk caches) to speed up repeated prompts. It also supports advanced inference techniques like continuous batching, speculative decoding, and prompt lookup decoding. For models too large for a single machine, it implements pipeline parallelism to distribute compute across multiple Macs via network connections.

Who it’s for

Developers and AI researchers using Apple Silicon hardware who want a high-performance, private, and compatible local inference server for multimodal AI models.

Highlights

  • Broad Model Support: Runs text LLMs, vision models, multimodal omni models, MoE, and hybrid SSMs.
  • Distributed Inference: Ability to split large models across multiple Macs using pipeline parallelism.
  • Multimodal Capabilities: Integrated support for image generation (Flux), image editing, text-to-speech (Kokoro), and speech-to-text (Whisper).
  • Advanced Quantization: Supports JANG adaptive mixed-precision to maintain higher quality at lower bit-widths.
  • Developer Tools: Includes a native macOS desktop app (MLX Studio) for model management, chat, and GGUF-to-MLX conversion.

Related

  • Project
    waybarrios/vllm-mlx

    vllm‑mlx is an Apple‑Silicon‑only inference server that runs LLMs, vision‑language models, audio models and embeddings on Metal via the MLX framework. It offers continuous batching, paged KV‑cache, SSD‑tiered cache and other vLLM‑style performance tricks, and exposes OpenAI‑compatible and Anthropic‑compatible REST APIs so existing client libraries work out‑of‑the‑box.

  • Project
    cubist38/mlx-openai-server

    An OpenAI-compatible API server for running local MLX models on Apple Silicon, supporting text, multimodal, image, and audio models.

  • Project
    ddalcu/mlx-serve

    A native Zig-based inference server for Apple Silicon that runs LLMs and multimodal models with OpenAI, Anthropic, and Ollama API compatibility.

  • Project
    jjang-ai/mlxstudio

    A native macOS app for Apple Silicon that provides a user-friendly interface to run LLMs, vision models, and image generation locally using the MLX framework.

  • Project
    jundot/omlx

    An optimized LLM inference server for Apple Silicon Macs featuring tiered KV caching, continuous batching, and a native macOS menu bar management app.