UbiquitousLearning/mllm

Fast Multimodal LLM on Mobile Devices

What it solves

mllm is a fast, lightweight multimodal LLM inference engine designed specifically for mobile and edge devices. It addresses the challenge of running large multimodal models on hardware with limited resources by providing a unified framework that bridges high-level model optimization algorithms and low-level hardware acceleration layers.

How it works

The engine functions as a central hub in the AI inference stack, connecting optimization techniques (like quantization, pruning, and speculative decoding) with hardware-specific runtimes (such as CUDA, CANN, and MLIR). It uses a mllm-convertor to ingest PyTorch and SafeTensors models, converting them into a proprietary mllm format for execution.

Key technical implementations include:

  • Hardware Backends: Support for Arm CPU, OpenCL GPU, QNN NPU, and Ascend NPU.
  • Android Architecture: A client-server design using an in-app Go server (mllm_server.aar) to decouple the UI from heavy inference computation.
  • Graph Extraction: A trace API that converts dynamic model execution into optimized static computational graphs for better serialization and deployment.
  • Optimization: Support for various quantization paths, including W4A16 and W8A8, with specific optimizations for Jetson Orin (using AWQ and Marlin GEMM).

Who it’s for

Developers and engineers building AI applications for edge devices, Android smartphones, and embedded systems (like NVIDIA Jetson) who need to run multimodal models with high throughput and low latency.

Highlights

  • Broad Hardware Support: Runs across Arm CPUs, OpenCL GPUs, and NPUs from Qualcomm (QNN) and Huawei (Ascend).
  • High Performance on Edge: Achieves up to 3.12x prefill speedup on AGX Orin 32GB for Qwen3-VL-2B W8A8 compared to llama.cpp.
  • Pythonic Model Authoring: Provides a high-level API for rapid model development and implementation.
  • Extensive Model Zoo: Supports a wide array of models including Qwen series, LLaMA, DeepSeek-OCR, and Phi-3-Vision.

Related

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    mlc-ai/mlc-llm

    MLC LLM is a universal deployment engine and ML compiler that enables high-performance native execution of large language models across diverse hardware and platforms.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.

  • Project
    vllm-project/vllm-ascend

    A hardware plugin that allows the vLLM inference engine to run on Huawei Ascend NPUs, supporting a wide range of LLM architectures.

  • Project
    baidu/vLLM-Kunlun

    A hardware plugin for vLLM that enables high-performance LLM inference on Kunlun XPU hardware with support for a wide range of models and quantization methods.