kyutai-labs/moshi

Moshi is a speech-text foundation model and full-duplex spoken dialogue framework. It uses Mimi, a state-of-the-art streaming neural audio codec.

What it solves

Moshi addresses the challenge of creating real-time, full-duplex spoken dialogue systems. It aims to provide a low-latency experience where the AI and user can speak and listen simultaneously, avoiding the awkward pauses typical of traditional turn-based speech systems.

How it works

Moshi is a speech-text foundation model that predicts two simultaneous audio streams: one for the AI's speech and one for the user's speech. To improve generation quality, it also predicts text tokens representing its own "inner monologue."

The system relies on two primary components:

  • Mimi: A streaming neural audio codec that compresses 24 kHz audio into a low-bandwidth representation with very low latency (80ms).
  • Transformers: A small Depth Transformer handles inter-codebook dependencies, while a large 7B-parameter Temporal Transformer manages temporal dependencies.

Who it’s for

  • Researchers: Those interested in speech-text foundation models and real-time dialogue.
  • Developers: People looking to integrate low-latency voice AI into applications, with support for PyTorch (research), MLX (macOS/iOS), and Rust (production).
  • End Users: Users who want to experience real-time, natural voice interaction via the provided web UI or CLI.

Highlights

  • Full-Duplex Dialogue: Supports simultaneous speaking and listening.
  • Ultra-Low Latency: Achieves a theoretical latency of 160ms and practical latency as low as 200ms on L4 GPUs.
  • Multi-Backend Support: Available in PyTorch, MLX (for Apple Silicon), and Rust (for production performance).
  • Integrated Codec: Includes Mimi, a high-performance streaming neural audio codec.

Related

  • Project
    moonshine-ai/moonshine

    An open source AI toolkit for building real-time, on-device voice agents and applications with low-latency speech-to-text and text-to-speech capabilities.

  • Project
    QuentinFuxa/WhisperLiveKit

    An ultra-low-latency, self-hosted speech-to-text pipeline that enables real-time transcription and translation using advanced simultaneous speech research and multiple model backends.

  • Project
    cobusgreyling/loop-engineering

    Loop Engineering is a CLI framework that lets you design, run, and score autonomous LLM‑agent loops (e.g., daily triage, PR babysitter) around a code repository. It provides ready‑made patterns, a STATE.md persistence file, safety checklists, and a progressive L1‑L3 scoring system to move from report‑only to fully unattended automation.

  • Project
    MemTensor/MemOS

    MemOS 2.0 is a Memory Operating System for LLM‑based AI agents. It offers a unified API to store, retrieve, edit, and delete multi‑modal memories (text, images, tool traces) organized as a graph. The system can run as a hosted cloud service (API‑key access) or be self‑hosted via Docker (Neo4j + Qdrant) or a local SQLite plugin. Ready‑made plugins integrate MemOS with OpenClaw, Hermes, and DeepSeek Harness. Benchmarks show it outperforms existing commercial memory products. Licensed under Apache 2.0.