jundot/omlx

What it solves

oMLX provides a dedicated, high-performance environment for running Large Language Models (LLMs) and Vision-Language Models (VLMs) locally on Apple Silicon Macs. It solves the common trade-off between ease of use and technical control by offering a managed server that handles memory, context caching, and model swapping, all accessible via a menu bar interface.

How it works

The software acts as a local server that manages model lifecycles and memory. It uses a tiered caching system: a "hot" tier keeps frequently used data in RAM for speed, while a "cold" tier offloads context to your SSD, allowing past conversation context to be reused even after a server restart. It supports continuous batching for multiple requests and provides an admin dashboard for monitoring, downloading models from HuggingFace, and configuring per-model settings like memory limits and aliases.

Who it’s for

It is designed for developers and power users on macOS who need to run local AI models for coding, tool-calling, or research. It is particularly useful for those who want to integrate local models with tools like Claude Code or need to manage multiple model types (text, vision, embedding, reranking) in a single, stable environment.

Highlights

• Tiered KV Caching: Automatically offloads context to SSD when RAM is full, enabling persistent memory across sessions.
• Native macOS Integration: Includes a SwiftUI menu bar app for monitoring and control, plus support for Apple Shortcuts.
• Broad Model Support: Handles LLMs, VLMs, OCR, embedding models, and rerankers with automatic detection.
• Admin Dashboard: A built-in web UI for real-time performance benchmarking, model management, and chat.
• API Compatibility: Provides a drop-in replacement for OpenAI and Anthropic APIs, making it easy to connect to existing AI tools.
• Efficient Resource Management: Features LRU (Least Recently Used) eviction, model pinning, and system-wide memory enforcement to prevent crashes.