trymirai/uzu

A high-performance inference engine for AI models

What it solves

uzu is a high-performance inference engine designed to let developers deploy AI models directly within their applications. It eliminates the need for external API calls for local models, which removes inference costs, ensures full data privacy, and removes network latency.

How it works

It provides a high-level API that abstracts the complexity of model downloading and inference configuration. The engine supports multiple languages (Rust, Python, Swift, and TypeScript) and is optimized for Apple devices by utilizing unified memory. It allows for both local model execution and integration with cloud models (like OpenAI) through a unified interface.

Who it’s for

Software developers building applications for desktop or mobile (specifically iOS and macOS) who want to integrate local AI capabilities without managing complex infrastructure.

Highlights

  • Multi-language support: Bindings for Rust, Python, Swift, and TypeScript.
  • Apple Silicon Optimization: Specifically leverages unified memory on Apple devices.
  • Unified API: A single interface for managing both local and cloud-based AI models.
  • Automatic Management: Handles model downloading and configuration automatically.
  • Traceable Computations: Ensures correctness by allowing computations to be traced against source-of-truth implementations.

Related

  • Project
    thu-pacman/chitu

    A production-grade high-performance inference engine that supports diverse hardware backends and scalable deployment for large language models.

  • Project
    Avarok-Cybersecurity/atlas

    A pure Rust and CUDA LLM inference engine that provides a lightweight, high-throughput OpenAI-compatible server without requiring Python or PyTorch.

  • Project
    openvinotoolkit/open_model_zoo

    A collection of optimized pre-trained deep learning models and tools for accelerating the development and deployment of high-performance inference applications.

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, agentic tool execution, and OpenAI/Anthropic compatible APIs.

  • Project
    zolotukhin/zinc

    A fast, local GGUF inference engine written in Zig that provides a command-line interface, browser chat, and OpenAI-compatible API for consumer GPUs.