lablup/mlxcel

High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon and NVIDIA CUDA systems.

What it solves

mlxcel is a high-performance inference engine designed to run Large Language Models (LLMs), Vision-Language Models (VLMs), embeddings, rerankers, and audio models on Apple Silicon, NVIDIA CUDA, and experimentally on AMD ROCm. It eliminates the need for a Python environment or interpreter layer in the request path, providing a native Rust-based runtime for production-oriented model serving.

How it works

The project implements a CLI and server in Rust that execute MLX SafeTensors checkpoints directly via native MLX C++ bindings. This architecture allows it to load checkpoints from HuggingFace (including mlx-community) without a conversion step. It supports advanced serving features like continuous batching, prompt-prefix caching, speculative decoding, and distributed inference.

Who it’s for

It is built for developers and operators who need a compact, native runtime for deploying AI models on Apple Silicon (primary target) or Linux/CUDA systems, as well as those seeking a high-performance, llama-server compatible API for local or production serving.

Highlights

  • Native Runtime: No Python required in the request path, simplifying deployment artifacts into native executables.
  • Broad Modality Support: Handles text generation, multimodal input (images/video), embeddings, reranking, ASR (transcription), and TTS (speech synthesis).
  • API Compatibility: Verified compatibility with llama-server and native support for OpenAI, Anthropic, and Vertex AI compatible routes.
  • Hardware Acceleration: Optimized for Apple Silicon, NVIDIA CUDA, and experimental support for AMD ROCm (RDNA 3.5).
  • Advanced Inference: Includes speculative decoding (MTP/DFlash), KV-cache compression, and a multi-model router with LRU eviction.
  • Model Surgery: Supports YAML-based load-time weight edits such as scaling, pruning, and interpolation.

Related

  • Project
    vllm-project/vllm-metal

    vLLM Metal is a plugin that enables the vLLM LLM serving system to run on Apple Silicon Macs using the MLX/Metal backend. It provides Metal‑optimized attention kernels (paged varlen, M5 NAX prefill) and supports large models such as Qwen‑3.8‑27B. Install via Homebrew or a curl script, then use the regular `vllm` command on macOS 15+.

  • Project
    ml-explore/mlx-swift

    MLX Swift provides Swift bindings for the MLX tensor library, enabling high‑performance machine‑learning (including LLMs, vision‑language models and diffusion) on Apple‑silicon GPUs (Metal) and Linux (CPU/CUDA). It offers core tensor ops, neural‑network layers, optimizers and example apps, and can be added to Xcode or SwiftPM projects.

  • Project
    ddalcu/mlx-serve

    A native Zig-based inference server for Apple Silicon that supports MLX and GGUF models with OpenAI, Anthropic, and Ollama API compatibility.

  • Project
    raullenchai/Rapid-MLX

    A high-performance LLM inference server for Apple Silicon that optimizes MLX for faster local inference and reliable tool-calling for coding agents.

  • Project
    ARahim3/mlx-dspark

    mlx‑d​spak is a Python package / macOS app that runs loss‑less speculative decoding (DSpark & DFlash) on Apple‑silicon Macs, giving 2‑5× faster text generation for many open‑source LLMs and exposing a local OpenAI‑compatible API.