sophgo/LLM-TPU

Run generative AI models in sophgo BM1684X/BM1688

LLM‑TPU – Deploy Large Language & Vision Models on SOPHGO TPUs

What it is – An open‑source toolkit from SOPHGO that lets you take popular generative AI models (LLMs such as Qwen, Llama, Phi, MiniCPM, etc. and multimodal vision‑language models) and run them on SOPHGO’s BM1684X, BM1688 or CV186X TPU chips with a single command.

Why it matters – Running big models on dedicated TPUs can give orders‑of‑magnitude speed‑up and lower power use compared with CPUs/GPUs, but the compilation and deployment steps are usually complex. LLM‑TPU bundles the whole pipeline – weight conversion, quantisation, bmodel generation and ready‑to‑run Python/C++ demos – into a streamlined workflow.

Key Features

Feature What it does
One‑click compilation llm_convert.py turns HuggingFace or GGUF checkpoints directly into a SOPHGO bmodel (the binary format the TPU understands).
Broad model catalog Dozens of models are pre‑packaged, from pure text LLMs (Qwen‑3, Llama‑3.2, Phi‑4, MiniCPM‑V) to multimodal VLMs (Mage‑VL, InternVL‑3, Qwen‑3.5‑VL, Gemma‑4) with image, video and audio support.
Quantisation support Works out‑of‑the‑box with AWQ/GPTQ/AutoRound int4 models; also offers optional --quantize flags for custom precision (w4bf16, bf16, etc.).
Dynamic & KV‑cache compilation --dynamic reduces latency for short inputs; --use_history_kv enables KV‑cache for multi‑turn conversations and long contexts.
Multi‑chip parallelism --num_device N spreads a model across several TPUs, allowing larger models and higher throughput.
Dual‑language demos Every model ships with both Python and C++ reference scripts, plus ready‑made pre‑compiled bmodels for instant testing.
Advanced utilities Shared prefill reuse, multi‑task weight sharing, optional model encryption, and LoRA‑aware compilation.

Quick Start (two commands)

git clone https://github.com/sophgo/LLM-TPU.git
cd LLM-TPU
./run.sh --model qwen3.5   # pulls a pre‑compiled bmodel and launches the demo

The script automatically downloads the appropriate bmodel (or compiles it if you have TPU‑MLIR set up) and starts an interactive chat that can accept image/video files via @<path>.

Supported Hardware

  • BM1684X – entry‑level TPU, good for 4‑8 B models.
  • BM1688 – higher‑end, handles 10‑30 B models with multi‑chip scaling.
  • CV186X – vision‑focused TPU, used for VLM inference.

Model Coverage (excerpt)

Category Example Models One‑click compile?
Text‑only LLMs Qwen‑3, Qwen‑2.5, Llama‑3.2, Phi‑4, MiniCPM‑4, DeepSeek‑R1 series ✅
Vision‑Language Mage‑VL, InternVL‑3, Qwen‑3.5‑VL, Qwen‑2.5‑VL, Gemma‑4 ✅
Audio / Video Qwen‑3.5 (audio), MiniCPM‑V‑4.6 (video), Gemma‑4 (audio) ✅
Legacy demos Baichuan‑2, ChatGLM‑3, RWKV, etc. (still in models/legacy/) —

Compilation Flow (example with Qwen3.5‑2B)

  1. Download weights – preferably an int4 AWQ/GPTQ checkpoint.
  2. Set up TPU‑MLIR – follow the linked repo or use the provided Docker image.
  3. Run the converter:
    llm_convert.py -m /path/Qwen3.5-2B-int4-AutoRound \
    -s 2048 --max_input_length 1024 \
    -c bm1684x -o qwen3.5_2b
    • -s = total KV‑cache length.
    • Add --use_history_kv and --chunk_length for long‑context, multi‑turn use.
  4. The output folder contains a *.bmodel file and a config directory ready for the demo scripts.

Getting the Most Accuracy

  • Prefer pre‑quantised AWQ/GPTQ/AutoRound checkpoints – they retain the original model’s quality after conversion.
  • If only FP16/FP32 weights exist, first quantise them with AutoAWQ or AutoGPTQ before running llm_convert.py.

Where to Learn More

  • FAQ – docs/FAQ.md
  • Paper – An MLIR‑Based Compilation Method for Large Language Models (arXiv:2607.15865)
  • TPU‑MLIR repo & docs – the compiler that actually produces the bmodel.
  • Video walkthroughs – Bilibili link in the README.

Contributing

Issues, pull‑requests and business inquiries are welcomed via the GitHub page or SOPHGO’s website.

License

Apache‑2.0 (see LICENSE).

TL;DR – LLM‑TPU is a practical, officially‑supported bridge that turns HuggingFace checkpoints into fast, TPU‑native inference for a wide range of text and multimodal models, with ready‑to‑run demos and extensive compile options.

Related

  • Project
    scaleapi/llm-engine

    An open-source engine for fine-tuning and serving large language models, providing tools to deploy models on hosted or self-managed Kubernetes infrastructure.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    ztxz16/fastllm

    A high-performance C++ inference engine for local LLM deployment that supports hybrid MoE offloading across GPU, CPU, NUMA, and disk.

  • Project
    qualcomm/GenieX

    An on-device Gen AI inference runtime for Qualcomm Snapdragon devices that allows users to run LLMs and VLMs locally using GGUF models or optimized Qualcomm AI Hub bundles.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.