NightMean/OlliteRT

What it solves

OlliteRT allows users to turn an Android device into a fully local, private LLM server. It eliminates the need for cloud subscriptions, API keys, and expensive GPU hardware by leveraging the mobile device's own CPU and GPU to host models that can be accessed over a local network.

How it works

The application uses Google's LiteRT-LM runtime to execute models on Android. It wraps this runtime in an OpenAI-compatible HTTP API server, allowing any software that can communicate with OpenAI's API (such as Open WebUI or Home Assistant) to send requests to the Android phone instead of a cloud service.

Who it’s for

It is designed for Android users (Android 12+, arm64-v8a) who want to run LLMs locally for privacy, cost-savings, or to repurpose old mobile hardware as low-power AI servers.

Highlights

• Broad Compatibility: Supports OpenAI and Anthropic compatible APIs, making it compatible with a wide range of existing AI clients.
• Multimodal Capabilities: Supports vision, audio, and "thinking" models (e.g., Gemma 4).
• Low Power Consumption: Consumes significantly less power (5-10W) compared to traditional GPU servers.
• Integrated Tooling: Includes built-in benchmarking, detailed activity logs, and Prometheus metrics for monitoring via Grafana.
• Flexible Model Management: Allows one-tap downloads from HuggingFace or importing custom .litertlm files.