jundot/omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

What it solves

Local LLM usage often forces a trade-off between convenience and control. Users frequently struggle with managing memory, keeping models loaded, or handling large contexts without exhausting system resources. oMLX provides a way to run LLMs, vision-language models, and embedding models on Mac hardware with high performance and easy management.

How it works

Running on Apple Silicon, oMLX uses a tiered KV cache system inspired by vLLM. It maintains a "hot" tier in RAM for fast access and a "cold" tier on the SSD to offload blocks when memory is full, allowing context to be reused across requests without recomputation. The system uses continuous batching to handle concurrent requests and includes a memory guard to prevent system-wide crashes by enforcing total memory limits.

Who it’s for

It is designed for macOS users (M1/M2/M3/M4) who want to run local AI models for tasks like coding (e.g., with Claude Code) or general chat, providing both a command-line interface for developers and a native macOS menu bar app for easy monitoring and control.

Highlights

  • Tiered KV Cache: Persists context between RAM and SSD to make long conversations practical.
  • Multi-Model Serving: Supports LLMs, VLMs, OCR, embeddings, and rerankers in a single server.
  • macOS Integration: Includes a native SwiftUI menu bar app for monitoring and an admin dashboard for management.
  • API Compatibility: Provides a drop-in replacement for OpenAI and Anthropic APIs.
  • Automated Management: Features LRU eviction, model pinning, and per-model idle timeouts to optimize memory usage.