dineshsoudagar/local-llms-on-android

What it solves

Pocket LLM allows users to run large language models (LLMs) entirely offline on Android devices, ensuring privacy and accessibility without relying on cloud services. It also transforms a phone into a local AI server that other devices on the same home network can access via a Web UI or an OpenAI-compatible API.

How it works

The app uses the ONNX and LiteRT backends to execute models on-device, leveraging GPU and NPU acceleration for performance. Users can download built-in models (like Gemma 4 or Qwen) or import their own .litertlm files. For document handling, it uses BM25 retrieval for RAG-like questioning and on-device OCR for scanned PDFs. Audio is handled via native model understanding or a multilingual Whisper fallback for transcription.

Who it’s for

Privacy-conscious users, AI enthusiasts, and developers who want a portable, offline AI assistant or a low-power local inference server for their home network.

Highlights

• Local LAN Server: Serve models to other devices via a password-protected Web UI or an OpenAI-compatible API.
• Multimodal Capabilities: Supports text, images (OCR and native vision), and audio (native understanding and Whisper transcription).
• Document Intelligence: Attach PDFs and text files for summaries or targeted questions using BM25 retrieval.
• Flexible Model Support: Includes built-in models like Gemma 4, Qwen3, and DeepSeek R1, with beta support for custom .litertlm imports.
• Privacy-First: All inference and data remain on the device with no telemetry.