trymirai/uzu

A high-performance inference engine for AI models

uzu is a high-performance inference engine that lets developers run AI models directly inside their apps — on-device, with no cloud round-trip. It's built for zero latency, full data privacy, and zero inference cost, and it's designed to work across Rust, Python, Swift, and TypeScript.

What it does:

  • Downloads models (e.g., alibaba:qwen3.5:0.8b:mirai:mirai-m:4) and runs chat sessions locally.
  • Provides a simple, high-level API: create an Engine, fetch a model, download it, start a ChatSession, and send messages.
  • Supports streaming replies, tool calls, structured output, and cloud models (e.g., GPT‑5 via OpenAI API) as alternatives.
  • Uses unified memory on Apple devices for efficient model loading.
  • Includes traceable computations to verify correctness against reference implementations.

How it works (from the examples):

  1. Create an EngineConfig and an Engine.
  2. Look up a model by identifier, download it with progress updates.
  3. Start a chat session, send a list of ChatMessages (system, user, etc.), and get replies — either all at once or as a stream.
  4. Reuse the same session for multiple requests; keep only one model loaded at a time to manage RAM.

Who it's for: Developers building apps (especially iOS/macOS) that need private, fast, offline AI — chatbots, assistants, or any feature requiring local inference. The project is MIT‑licensed and actively maintained, with bindings for multiple languages and a documented API.

Related

  • Project
    thu-pacman/chitu

    A production-grade high-performance inference engine that supports diverse hardware backends and scalable deployment for large language models.

  • Project
    openvinotoolkit/open_model_zoo

    A collection of optimized pre-trained deep learning models and tools for accelerating the development and deployment of high-performance inference applications.

  • Project
    zolotukhin/zinc

    ZINC is a native binary that runs GGUF language models on local GPUs, providing a command-line interface, a browser-based chat, and an OpenAI-compatible API.

  • Project
    incoai/splash

    Splash is a high-performance local inference engine for Apple Silicon that accelerates LLM execution using specialized Metal kernels and speculative decoding, providing an OpenAI-compatible API for agents and chat applications.

  • Project
    EricLBuehler/mistral.rs

    mistral.rs is a Rust‑based engine for fast, flexible local LLM inference. It supports many model formats (Hugging Face, GGUF), automatic hardware‑aware quantization, multimodal inputs, and built‑in agentic features (tool calling, Python/shell execution, web search). It can serve OpenAI‑compatible and Anthropic‑compatible APIs, offers a web UI, and provides Rust and Python SDKs for embedding.