trymirai/uzu
A high-performance inference engine for AI models
What it solves
uzu is a high-performance inference engine designed to let developers deploy AI models directly within their applications. It eliminates the need for external API calls, which removes inference costs, ensures full data privacy, and provides zero-latency responses by running models locally.
How it works
uzu provides a high-level API across multiple languages (Rust, Python, TypeScript, and Swift) to manage the entire model lifecycle. It handles model discovery, automatic downloading, and inference configuration. On Apple devices, it optimizes performance by utilizing unified memory. It also supports a hybrid approach, allowing developers to switch between local on-device models and cloud-based models (such as OpenAI) using a unified interface.
Who it’s for
It is built for application developers who want to integrate LLMs into their software—specifically for iOS, macOS, and cross-platform apps—without relying on cloud infrastructure or incurring per-token costs.
Highlights
- Multi-language Support: Native bindings for Rust, Python, TypeScript, and Swift.
- On-Device Optimization: Specifically leverages unified memory on Apple hardware for efficiency.
- Privacy and Cost: Local execution ensures data never leaves the device and eliminates API fees.
- Structured Output: Supports the use of Grammars and JSON schemas to force models to return valid, structured data.
- Hybrid Flexibility: Unified API for both local on-device inference and cloud-model integration.