cubist38/mlx-openai-server

A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

What it solves

It provides an OpenAI-compatible API server for running various local AI models on Apple Silicon using the MLX framework. This allows users to use familiar OpenAI SDKs and clients to interact with models running locally on their Mac, eliminating the need for external APIs.

How it works

The server acts as a bridge between OpenAI-style API requests and several MLX-based backends. It supports multiple model types including text (lm), multimodal (vlm), image generation and editing (mflux), embeddings, and audio transcription (whisper). It can serve a single model via the command line or multiple models simultaneously using a YAML configuration file, where each model runs in its own subprocess to isolate the Metal runtime state.

Who it’s for

Developers and AI researchers using macOS on Apple Silicon who want to deploy local models with a standardized API interface for integration into existing applications or agents.

Highlights

  • Broad Model Support: Serves text, vision-language models, image generation/editing, embeddings, and Whisper transcription.
  • OpenAI Compatibility: Implements standard endpoints like /v1/chat/completions, /v1/images/generations, and /v1/embeddings.
  • Performance Tuning: Includes options for continuous batching, speculative decoding, and KV cache quantization to optimize memory and throughput.
  • Memory Management: Provides detailed guidance and configuration options to prevent Metal Out-of-Memory (OOM) errors on macOS.
  • Multi-Model Hosting: Supports loading multiple models via YAML config, including an "on-demand" loading feature to save memory.

Related

  • Project
    jjang-ai/vmlx

    A self-hosted inference server for Apple Silicon that provides OpenAI and Anthropic compatible APIs for running LLMs, VLMs, and image generation models locally.

  • Project
    raullenchai/Rapid-MLX

    A high-performance local AI engine for Apple Silicon that provides OpenAI and Anthropic-compatible APIs for running LLMs, audio, and video models natively on M-series Macs.

  • Project
    Arthur-Ficial/apfel

    A UNIX tool and OpenAI-compatible server that exposes Apple Silicon Macs' built-in FoundationModels LLM for 100% on-device AI inference.

  • Project
    scouzi1966/maclocal-api

    A native Swift AI infrastructure for Apple Silicon that turns a Mac into a private, OpenAI-compatible AI server supporting MLX and Apple Foundation Models.

  • Project
    waybarrios/vllm-mlx

    vllm‑mlx is an Apple‑Silicon‑only inference server that runs LLMs, vision‑language models, audio models and embeddings on Metal via the MLX framework. It offers continuous batching, paged KV‑cache, SSD‑tiered cache and other vLLM‑style performance tricks, and exposes OpenAI‑compatible and Anthropic‑compatible REST APIs so existing client libraries work out‑of‑the‑box.