lablup/mlxcel

High-performance LLM/VLM inference runtime and server for Apple Silicon / NVIDIA CUDA devices

What it solves

mlxcel is a high-performance inference runtime and server designed to eliminate the need for a Python environment when deploying Large Language Models (LLMs) and Vision-Language Models (VLMs). It provides a native, single-process execution environment that simplifies deployment, packaging, and service supervision while maintaining performance parity with Python-based MLX references.

How it works

Implemented in Rust, mlxcel uses native MLX C++ bindings to execute models. It supports Apple Silicon, NVIDIA CUDA-compatible devices, and experimental OpenXLA-compatible devices. The runtime handles model loading, scheduling, and inference within a single native process, allowing it to run models directly from mlx-community checkpoints without requiring a conversion step.

Who it’s for

It is intended for developers and operators who want to deploy local or small-cluster inference servers with minimal overhead, those migrating from llama.cpp (due to its OpenAI-compatible API and similar flag structure), and users of Apple Silicon or NVIDIA GPUs.

Highlights

  • Native Performance: Achieves near-parity with mlx-lm and mlx-vlm decode throughput, with significantly faster short-prompt text prefill.
  • Broad Model Support: Supports a wide array of text and vision-language families including Llama, Qwen, Gemma, Phi, Mistral, DeepSeek, and others.
  • OpenAI Compatible: Provides an OpenAI-compatible HTTP API for /v1/chat/completions, /v1/completions, and /v1/responses.
  • Production-Ready Features: Includes continuous batching, prompt-prefix caching, speculative decoding, and KV-cache compression.
  • Model Surgery: Offers first-class YAML load-time model surgery (scale, add, prune, replace, interpolate) for weight-space changes without retraining.
  • Distributed Inference: Implements tensor and pipeline parallelism for selected model families.

Related

  • Project
    ddalcu/mlx-serve

    A native Zig-based inference server for Apple Silicon that supports MLX and GGUF models with OpenAI, Anthropic, and Ollama API compatibility.

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    Avarok-Cybersecurity/atlas

    A pure Rust LLM inference engine that uses hardware-specific kernels to maximize local inference speed and stability.

  • Project
    jjang-ai/vmlx

    A self-hosted inference server for Apple Silicon that provides OpenAI and Anthropic compatible APIs for running LLMs, VLMs, and image generation models locally.

  • Project
    waybarrios/vllm-mlx

    vllm‑mlx is an Apple‑Silicon‑only inference server that runs LLMs, vision‑language models, audio models and embeddings on Metal via the MLX framework. It offers continuous batching, paged KV‑cache, SSD‑tiered cache and other vLLM‑style performance tricks, and exposes OpenAI‑compatible and Anthropic‑compatible REST APIs so existing client libraries work out‑of‑the‑box.