modal-labs/quillman

A voice chat app

What it solves

QuiLLMan is a voice chat application that enables near-instantaneous, human-like speech interaction. It solves the latency issues typically associated with voice AI by using bidirectional streaming and efficient audio compression.

How it works

The app uses a speech-to-speech language model (Moshi) and a streaming encoder/decoder (Mimi) to continuously listen, plan, and respond. It employs a bidirectional websocket streaming architecture and the Opus audio codec to compress audio data across the network, ensuring low-latency responses.

Who it’s for

Developers looking for a starting point or playground for building language model-based voice applications.

Highlights

  • Speech-to-Speech: Uses the Moshi model for continuous listening and responding.
  • Low Latency: Achieves near-instantaneous response times via bidirectional websockets and Opus compression.
  • Full Stack: Includes a React frontend and a FastAPI backend hosted on Modal.
  • Serverless Deployment: Designed for easy deployment to Modal, allowing the application to scale to zero when not in use.

Related

  • Project
    kyutai-labs/unmute

    A system that enables text LLMs to listen and speak by wrapping them in low-latency speech-to-text and text-to-speech models.

  • Project
    QuentinFuxa/WhisperLiveKit

    An ultra-low-latency, self-hosted speech-to-text pipeline that enables real-time transcription and translation using advanced simultaneous speech research and multiple model backends.

  • Project
    dimastatz/whisper-flow

    A real-time transcription service that enables streaming speech-to-text using OpenAI Whisper, providing low-latency results instead of batch processing.

  • Project
    datascale-ai/opentalking

    An open-source orchestration framework for real-time digital-human conversations, integrating LLMs, TTS, STT, and video rendering via WebRTC.

  • Project
    KoljaB/RealtimeTTS

    A Python text-to-speech library designed for low-latency audio generation from strings, generators, and LLM token streams, supporting a wide variety of local and cloud TTS engines.