OpenBMB/MiniCPM-o-Demo

Official PyTorch+CUDA Full-functional Web Demo for MiniCPM-o 4.5

What it solves

This project provides a demo system for MiniCPM-o 4.5, an omnimodal model designed for real-time, full-duplex interaction. It solves the challenge of demonstrating the model's ability to see, listen, and speak simultaneously without mutual blocking, providing a fluid conversation experience that surpasses traditional turn-based AI interactions.

How it works

The system uses a PyTorch + CUDA inference backend (with an optional C++ backend via llama.cpp) and a lightweight frontend-backend architecture. It employs a time-division multiplexing (TDM) mechanism to sync input and output streams on a millisecond timeline, allowing the model to process continuous video and audio streams while generating concurrent text and speech outputs. A proactive interaction mechanism allows the model to monitor streams at a 1Hz frequency to decide when to initiate speech independently.

Who it’s for

  • Developers looking to implement or test real-time multimodal AI interfaces.
  • Researchers interested in end-to-end omnimodal architectures and full-duplex streaming.
  • Users wanting to experience high-performance vision-language and speech capabilities on local GPUs or PCs.

Highlights

  • Full-Duplex Interaction: Supports real-time omnimodal live streaming where the model can see, listen, and speak at the same time.
  • Proactive Interaction: The model can autonomously initiate reminders or comments based on its continuous understanding of the scene.
  • Omnimodal Capabilities: Combines strong visual performance (surpassing GPT-4o in some benchmarks), bilingual real-time speech, and state-of-the-art OCR.
  • Flexible Deployment: Supports PyTorch, C++ (llama.cpp), and various quantization formats (int4, GGUF) for deployment across GPUs, PCs, and MacBooks.

Related

  • Project
    OpenBMB/MiniCPM-V

    A series of pocket-sized multimodal LLMs designed for efficient on-device deployment, enabling high-performance image, video, and real-time omnimodal interaction on mobile phones.

  • Project
    jingyaogong/minimind-o

    MiniMind‑O is a genuine open‑source, tiny (≈0.1 B) end‑to‑end omni model that jointly handles text, speech and images as inputs and produces text and streamed speech outputs. It provides full PyTorch code, training scripts, mini/full datasets, pretrained weights, voice‑cloning prompts, and demo interfaces (CLI, Gradio, phone‑mode UI). The repository is clearly an AI/ML project, not an off‑topic list.

  • Project
    OpenSQZ/MiniCPM-V-CookBook

    A comprehensive implementation guide and deployment cookbook for the MiniCPM series, enabling the integration of vision, speech, and text AI capabilities across edge devices and enterprise servers.

  • Dispatch
    Qwen2.5-Omni Release: End-to-End Multimodal Model for Real-Time Interaction

    Qwen has released Qwen2.5-Omni, an end-to-end multimodal model capable of processing text, images, audio, and video to generate real-time streaming text and natural speech responses.

  • Project
    OpenBMB/MiniCPM-V-Apps

    A collection of on-device mobile demos for the MiniCPM-V family of multimodal models, enabling local AI inference on iOS, Android, and HarmonyOS.