shubham0204/SmolChat-Android

Running any GGUF SLMs/LLMs locally, on-device in Android

What it solves

SmolChat provides a native Android interface for running small language models (SLMs) directly on your mobile device. It removes the need for cloud-based AI services, allowing users to perform local inference while maintaining control over their data and model parameters.

How it works

The application leverages the llama.cpp C++ library, which is compiled for Android using the NDK. It uses JNI (Java Native Interface) bindings to bridge the gap between the Kotlin-based Android UI and the C++ inference engine. Users can import GGUF-format models, configure system prompts, and adjust inference settings like temperature and min-p directly within the app.

Who it’s for

This project is for Android developers and AI enthusiasts who want to experiment with running local, privacy-focused language models on mobile hardware without relying on external APIs.

Highlights

  • Runs SLMs locally on-device using the llama.cpp engine.
  • Supports custom GGUF model imports and configurable inference parameters.
  • Includes a dedicated module for managing chat history and system prompts.
  • Features built-in Markdown rendering and code syntax highlighting for model responses.

Related

  • Project
    TheStageAI/TheWhisper

    TheWhisper is an open‑source, high‑performance speech‑to‑text system built on fine‑tuned Whisper‑large models. It offers streaming, low‑latency inference on NVIDIA GPUs (via TheStage AI’s ElasticModels) and on Apple Silicon (CoreML/MLX), with Python APIs, Electron demos, and benchmarks showing ~220 tokens/s on an L40 GPU and ~2 W power on Macs.

  • Project
    mosecorg/mosec

    Mosec is a Rust‑backed, Python‑friendly model‑serving framework that offers dynamic batching, multi‑stage pipelines, and built‑in metrics. Install via pip/conda, define a `Worker` subclass with a `forward` method, and launch a `Server` to expose any ML model (e.g., Stable Diffusion) as a fast HTTP API.

  • Project
    toby-bridges/api-relay-audit

    API Relay Audit is a zero‑dependency Python CLI that locally tests third‑party AI API relays (LLM proxies) for hidden prompt injection, context truncation, model substitution, stream anomalies, and Web3‑wallet safety, outputting a structured Markdown or JSON risk report.

  • Project
    guidance-ai/llguidance

    A high-performance engine for constrained decoding that forces LLMs to generate text adhering to specific grammars or JSON schemas, ensuring reliable structured outputs.