jundot/omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

What it solves

oMLX is a high-performance LLM inference server specifically optimized for Apple Silicon Macs. It addresses the trade-off between convenience and control by providing a managed environment where users can pin models in memory, auto-swap heavy models on demand, and manage the entire server from a native macOS menu bar app.

How it works

The project uses a tiered KV (Key-Value) cache system—inspired by vLLM—that splits storage between a "hot" RAM tier for fast access and a "cold" SSD tier for persisting context across restarts or context changes. It employs continuous batching via mlx-lm to handle concurrent requests and provides an OpenAI/Anthropic-compatible API for easy integration with external tools. For advanced users, it supports experimental multi-Mac inference, allowing a single model to be split across multiple Macs using pipeline ranks over Ring or Thunderbolt RDMA.

Who it’s for

Developers and power users on macOS who want to run local LLMs, VLMs, and embedding models with professional-grade memory management and high throughput, particularly those using coding assistants like Claude Code.

Highlights

  • Tiered KV Caching: Persists conversation context across RAM and SSD to avoid recomputing prompts.
  • Native macOS Integration: Includes a Swift-based menu bar app for monitoring and server control, avoiding the need for a terminal.
  • Multi-Model Serving: Supports simultaneous loading of LLMs, VLMs, OCR models, and rerankers with LRU eviction and model pinning.
  • Admin Dashboard: A comprehensive web UI for model downloading (from HuggingFace), benchmarking, and per-model configuration.
  • API Compatibility: Drop-in replacement for OpenAI and Anthropic APIs, including support for tool calling and MCP (Model Context Protocol) integration.

Related

  • Project
    jjang-ai/vmlx

    A self-hosted inference server for Apple Silicon that provides OpenAI and Anthropic compatible APIs for running LLMs, VLMs, and image generation models locally.

  • Project
    waybarrios/vllm-mlx

    vllm‑mlx is an Apple‑Silicon‑only inference server that runs LLMs, vision‑language models, audio models and embeddings on Metal via the MLX framework. It offers continuous batching, paged KV‑cache, SSD‑tiered cache and other vLLM‑style performance tricks, and exposes OpenAI‑compatible and Anthropic‑compatible REST APIs so existing client libraries work out‑of‑the‑box.

  • Project
    SharpAI/SwiftLM

    A native Swift inference server for Apple Silicon that serves MLX models via an OpenAI-compatible API, featuring SSD expert streaming and KV cache compression.

  • Project
    ddalcu/mlx-serve

    A native Zig-based inference server for Apple Silicon that runs LLMs and multimodal models with OpenAI, Anthropic, and Ollama API compatibility.

  • Project
    LMCache/LMCache

    A KV cache management layer for LLM inference that reduces TTFT and improves throughput by making KV caches persistent and reusable across requests and engines.