tc-mb/llama.cpp-omni

What it solves

llama.cpp-omni provides a high-performance, local inference engine for omni-modal models (like MiniCPM-o 4.5). It enables real-time, full-duplex voice and video interaction on consumer hardware by optimizing the entire pipeline—from vision and audio encoding to text-to-speech synthesis—into a single, efficient C++ framework.

How it works

The engine builds on the llama.cpp ecosystem, splitting the complex MiniCPM-o architecture into modular GGUF files. It uses a Time-Division Multiplexing (TDM) mechanism to synchronize parallel streams of video, audio, and text. By processing inputs in 1-second chunks and using asynchronous queues for generation, it allows the model to "listen" and "speak" simultaneously without blocking, enabling proactive, human-like conversation.

Who it’s for

This project is for developers and researchers who want to run advanced, multimodal AI agents locally on their own hardware (Windows, Linux, or macOS) without relying on cloud APIs. It is particularly useful for those building real-time voice assistants or interactive video-call applications.

Highlights

• Full-Duplex Streaming: Supports simultaneous input (video/audio) and output (speech/text) for fluid, real-time interaction.
• High Performance: Achieves sub-650ms time-to-first-token latency on consumer hardware like RTX 4090 or Apple M4 Max.
• Modular Architecture: Separates vision (VPM), audio (APM), language (LLM), and speech (TTS) components into GGUF modules for flexible deployment.
• Proactive Interaction: Enables the model to monitor streams continuously and decide when to speak spontaneously.
• Ecosystem Compatibility: Fully compatible with the llama.cpp toolchain and supports common quantization formats (F16, Q8_0, Q4_K_M).