OpenAI GPT-Live: Engineering a Real-time Voice AI System

OpenAI has developed GPT-Live, a third-generation voice system designed to eliminate the sluggishness of traditional turn-based voice AI. By replacing turn detectors with a full-duplex voice model that can listen and speak simultaneously, GPT-Live achieves sub-second responsiveness and a more natural conversational rhythm.

Transitioning from Turn-Based to Streaming Architecture

GPT-Live moves away from cascaded systems—where speech-to-text, LLMs, and text-to-speech ran in series—and previous speech-to-speech models that still relied on turn detectors to trigger inference.

In the new architecture, the voice model is in direct control of the conversation. Audio flows continuously in and out of the model, while deeper reasoning and tool use are handled asynchronously. This ensures the primary media loop remains uninterrupted, decoupling the "talking" from the "thinking."

Engineering for Continuous Inference and Low Latency

To maintain a seamless media loop and prevent audible artifacts, OpenAI implemented several systemic optimizations:

Dedicated Media Path and Language Shift

OpenAI separated the media flow from application and business logic. Audio moves on a dedicated fast path, while delegation and tool use occur behind an asynchronous RPC boundary. To improve frame delivery smoothness, the media frontend and inference logic were rewritten in Go, replacing a previous Python asyncio implementation. This change resulted in the p95 latency matching the previous system's p50.

Transport and Protocol Optimizations

WebRTC serves as the transport foundation, allowing the system to handle packet loss and clock drift. To further reduce startup latency, OpenAI introduced two key innovations:

  • WARP: A set of open specifications designed to collapse the transport handshake and reduce network round trips.
  • Instant Connect: A mechanism that negotiates SDP parameters ahead of time, allowing a session to start with a single UDP packet.

Stateful Inference and Context Management

To support long-running conversations without interruption, OpenAI developed a seamless handoff mechanism. When a model instance needs to be replaced or the context needs compaction (to fit within model limits), the system warms a replacement instance in parallel, prefills it with the necessary context, and cuts over only when the new instance is ready. This prevents the KV cache rebuild from causing audible delays.

Asynchronous Delegation to Frontier Models

GPT-Live integrates with frontier models like GPT-5.5 for complex reasoning and search without blocking the live voice path.

Optimizing the Delegation Loop

To ensure delegated results return quickly enough to be useful, OpenAI optimizes the entire loop—routing, prompt processing, inference, and tool calls. This includes creating an inference session for the frontier model and prefilling it with conversation context as soon as the voice session starts, using stable session affinity and prompt caching to minimize latency.

Deriving Discrete Turns from Continuous Speech

Because surrounding systems (such as the UI and safety infrastructure) require discrete messages, the application server uses partial transcripts and timing signals to infer speaker turns. The system maintains a speculative view for the UI and an authoritative record for analytics, allowing the voice path to remain continuous while providing a stable record of the exchange.

Production Testing and Scaling

OpenAI validated GPT-Live using "silent tests," routing a share of production ChatGPT Voice sessions to the new system in read-only mode. This process revealed several critical insights:

  • Capacity Planning: Capacity is determined by concurrent sessions and the ability to keep every frame on schedule, rather than just GPU throughput.
  • Geographic Distribution: End-to-end responsiveness is heavily dependent on routing sessions to regional capacity to minimize distance-based delay.
  • Lifecycle Failures: Long-running sessions and reconnects exposed memory pressure and state restoration issues that were not apparent in short load tests.

This architecture now powers ChatGPT Voice capabilities, including computer control and agent coordination in the desktop app, and will serve as the foundation for the upcoming GPT-Live API.

Sources