modal-labs/quillman

A voice chat app

What it solves

QuiLLMan is a voice chat application that enables near-instantaneous, human-like speech interaction. It solves the latency issues typically associated with voice AI by using bidirectional streaming and efficient audio compression.

How it works

The app uses a speech-to-speech language model (Moshi) and a streaming encoder/decoder (Mimi) to continuously listen, plan, and respond. It employs a bidirectional websocket streaming architecture and the Opus audio codec to compress audio data across the network, ensuring low-latency responses.

Who it’s for

Developers looking for a starting point or playground for building language model-based voice applications.

Highlights

  • Speech-to-Speech: Uses the Moshi model for continuous listening and responding.
  • Low Latency: Achieves near-instantaneous response times via bidirectional websockets and Opus compression.
  • Full Stack: Includes a React frontend and a FastAPI backend hosted on Modal.
  • Serverless Deployment: Designed for easy deployment to Modal, allowing the application to scale to zero when not in use.

Related

  • Project
    kyutai-labs/unmute

    A system that enables text LLMs to listen and speak by wrapping them in low-latency speech-to-text and text-to-speech models.

  • Project
    QuentinFuxa/WhisperLiveKit

    An ultra-low-latency, self-hosted speech-to-text pipeline that enables real-time transcription and translation using advanced simultaneous speech research and multiple model backends.

  • Project
    dimastatz/whisper-flow

    Whisper Flow is a FastAPI‑based service that turns OpenAI’s Whisper model into a real‑time streaming speech‑to‑text engine. Clients stream 16 kHz mono PCM audio over a WebSocket; the server buffers audio into tumbling windows, runs Whisper, and pushes back partial and final JSON transcripts with sub‑second latency (≈275 ms) and ~7 % WER on an M1 Mac. It can be run locally, via Docker, or installed as a Python package for integration.

  • Project
    datascale-ai/opentalking

    An open-source orchestration framework for real-time digital-human conversations, integrating LLMs, TTS, STT, and video rendering via WebRTC.

  • Project
    KoljaB/RealtimeTTS

    A Python library that provides a unified, low-latency interface for streaming text-to-speech, designed to integrate seamlessly with LLM token streams and various local or cloud-based audio engines.