OpenBMB/MiniCPM-V

MiniCPM‑V 4.7 & MiniCPM‑o 4.5 – On‑device multimodal LLMs

What it is – An open‑source family of multimodal large language models (LLMs) released by the Tsinghua NLP group (THUNLP) and ModelBest.

MiniCPM‑V 4.7: vision‑language models (image, multi‑image, video, document) in two sizes – a 35 billion‑parameter Mixture‑of‑Experts (MoE) variant (35B‑A3B) and a lightweight 1 billion‑parameter version.
MiniCPM‑o 4.5: a 9 billion‑parameter “omni‑modal” model that can ingest video + audio continuously while generating text and speech in real time.

Why it matters

First open‑source MoE that runs smoothly on a phone – thanks to the Lookahead Expert Scheduling (LES) technique, the 35B‑A3B model can decode >13 tokens / s on an iPhone 17 Pro Max using only ~3 GB of RAM.
Performance‑efficiency frontier – both the 35B‑A3B and 1B variants beat comparable commercial models (Qwen 3.6 35B‑A3B, Qwen 3.5 0.8B) in overall benchmark scores while using less inference time.
Full‑duplex omni‑modal interaction – MiniCPM‑o 4.5 can watch video, listen to audio, and speak back at the same time, enabling live, proactive assistants (e.g., reminders triggered by what the camera sees).
Cross‑platform deployment – supported on iOS, Android, HarmonyOS and on a wide range of CPUs/NPUs (Intel, AMD, Arm, Rockchip, Houmo).
Self‑Replay Fine‑Tuning (SRFT) – a low‑cost fine‑tuning recipe that adds new abilities while minimizing forgetting of existing skills.

Core capabilities

Model	Vision	Speech	Language	Interaction mode
MiniCPM‑V 4.7 (35B‑A3B / 1B)	Image, multi‑image, video, document understanding (mixed 4×/16× visual token compression)	–	Text generation, instruction following, reasoning	Instruct & reasoning modes (offline inference)
MiniCPM‑o 4.5 (9B)	Real‑time visual understanding (approaches Gemini 2.5 Flash)	Bilingual speech‑to‑text & text‑to‑speech, voice cloning	Text generation	Full‑duplex live streaming of video + audio input while outputting text + speech

Getting started (quick links)

Model weights – Hugging Face repos: MiniCPM‑V‑4.7‑35B‑A3B, MiniCPM‑V‑4.7‑1B, MiniCPM‑o‑4.5.
Inference – Use the provided Transformers‑compatible scripts (docs/...#inference-with-transformers) or the llama.cpp‑omni runtime for on‑device MoE inference.
Fine‑tuning – Guides for SRFT and LLaMA‑Factory/Swift training are in docs/deployment_en.md.
Live demos – Hugging Face Space for MiniCPM‑V 4.7 and a web demo for MiniCPM‑o 4.5.
API – REST‑style API described in docs/api.md (available for both models).
Cookbook – Community recipes for deployment, prompt engineering, and tool integration: https://github.com/OpenSQZ/MiniCPM-V-CookBook.

Who might use it

Mobile AI developers who need a powerful vision‑language model that fits on‑device (privacy‑first apps, AR, offline OCR, document analysis).
Real‑time assistants that must see, hear, and speak simultaneously – e.g., smart home hubs, robot companions, live‑stream moderation tools.
Researchers exploring efficient MoE inference, multimodal token compression, or full‑duplex interaction pipelines.
Educators & hobbyists looking for a state‑of‑the‑art multimodal LLM that can be run locally without cloud costs.

Community & ecosystem

Discord, WeChat, Feishu groups for support and collaboration.
Third‑party projects built on MiniCPM (digital‑human chat, OCR pipelines, ComfyUI integration).
Paper references – arXiv pre‑prints describing the models and the LES technique (see citation block).

License & usage

Code and model weights are released under the Apache‑2.0 license.
A separate model‑specific license governs commercial use (see docs/project_en.md#model-license).

TL;DR

MiniCPM‑V 4.7 brings a 35 B‑parameter MoE vision‑language model to phones with real‑time speeds, while MiniCPM‑o 4.5 delivers the first open‑source 9 B omni‑modal model that can watch, listen, and speak at the same time. Both are fully open‑source, cross‑platform, and come with documentation, demos, and fine‑tuning recipes.