UbiquitousLearning/mllm

Fast Multimodal LLM on Mobile Devices

What it solves

mllm is a fast and lightweight multimodal LLM inference engine designed specifically for mobile and edge devices. It addresses the challenge of running large vision-language models (VLMs) and LLMs locally on hardware with limited resources, such as smartphones and embedded systems, without relying on cloud servers.

How it works

The engine acts as a bridge between high-level optimization algorithms and low-level hardware execution layers. It uses a conversion tool (mllm-convertor) to ingest PyTorch and SafeTensors models, quantizing and converting them into a proprietary mllm format for execution.

Key technical components include:

  • Hardware Backends: Support for Arm CPU, OpenCL GPU, QNN NPU, and Ascend NPU.
  • Optimizations: Implementation of quantization (e.g., W4A16, W8A8), pruning, and speculative execution to reduce memory and compute requirements.
  • Architecture: On Android, it employs a Client-Server architecture using an In-App Go Server to decouple the UI from heavy inference tasks.
  • Graph Execution: A trace API allows for computational graph extraction, converting dynamic execution into optimized static representations for NPU integration.

Who it’s for

Developers and engineers building on-device AI applications for Android, Jetson Orin, and other edge devices who need high-performance local inference for multimodal models.

Highlights

  • Broad Hardware Support: Runs on Arm CPUs, OpenCL GPUs, and NPUs from Qualcomm (QNN) and Huawei (Ascend).
  • High Performance on Edge: Demonstrates significant prefill speedups (up to 3.12x on Jetson Orin for specific models) compared to llama.cpp.
  • Extensive Model Zoo: Supports a wide range of models including Qwen series (Qwen3, Qwen2-VL), MiniCPM, Llama 3, and DeepSeek-OCR.
  • Developer Friendly: Provides a Pythonic API for model authoring and a C++ SDK for integration.

Related

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without server-side processing.

  • Project
    mlc-ai/mlc-llm

    MLC LLM is a universal deployment engine and ML compiler that enables high-performance native execution of large language models across diverse hardware and platforms.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    ztxz16/fastllm

    FastLLM is a C++‑based, PyTorch‑free inference engine for large language models. It supports dense and MoE models, hybrid GPU/CPU/NUMA/disk deployment, multi‑GPU tensor parallelism, speculative decoding (MTP/DSpark/DFlash2), many quantization formats, and offers a Python CLI, web UI, and OpenAI‑compatible API server. Install via `pip install ftllm`, run a model with `ftllm run`, or launch a service with `ftllm server`. Extensive docs cover model‑specific guides, performance tuning, and community support.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.