OpenAI Low-Latency Voice AI Infrastructure

OpenAI has implemented a split relay-plus-transceiver architecture to solve the scaling challenges of delivering real-time voice AI. By decoupling packet routing from protocol termination, OpenAI can maintain the low-latency requirements of natural conversation while operating within a standard Kubernetes environment.

The Challenge of Scaling WebRTC on Kubernetes

Standard WebRTC implementations often rely on a one-port-per-session model, which creates significant operational hurdles when deployed at OpenAI's scale (900 million+ weekly active users). This model introduces three primary constraints:

• Port Exhaustion: Managing tens of thousands of public UDP ports per service increases complexity in load balancer configuration, firewall policies, and rollout safety.
• Security Risks: Large UDP port ranges expand the externally reachable surface area, making network policies harder to audit.
• Autoscaling Friction: Kubernetes pods are frequently rescheduled; requiring each pod to reserve and advertise a large stable port range makes elasticity brittle.

While single-port-per-server designs reduce the port count, they struggle with "state stickiness." Because ICE (Interactive Connectivity Establishment) and DTLS (Datagram Transport Layer Security) are stateful, packets for a specific session must consistently reach the process that owns that session to avoid connection failure.

The Relay + Transceiver Architecture

To resolve these conflicts, OpenAI moved away from the Selective Forwarding Unit (SFU) model—typically used for multiparty calls—in favor of a transceiver model. In this architecture, the WebRTC edge service (the transceiver) terminates the client connection and converts media into internal protocols for inference and orchestration.

Split Routing and Termination

The core of this design is the separation of the Relay (routing) and the Transceiver (termination):

• The Relay: A lightweight UDP forwarding layer with a small public footprint. It does not decrypt media, run ICE state machines, or negotiate codecs. It simply reads packet metadata to determine the destination and forwards the packet.
• The Transceiver: The stateful WebRTC endpoint. It owns the full session state, including ICE connectivity checks, the DTLS handshake, SRTP encryption keys, and the session lifecycle.

First-Packet Routing via ICE ufrag

To avoid pausing for external lookups on the first packet, OpenAI utilizes the ICE username fragment (ufrag). During session setup, the transceiver generates a server-side ufrag containing routing metadata.

When the client sends its first STUN (Session Traversal Utilities for NAT) binding request, the relay parses the ufrag, decodes the routing hint, and forwards the packet to the owning transceiver. Once the route is established, subsequent DTLS, RTP, and RTCP packets flow based on an in-memory session map, supplemented by a Redis cache to recover mappings if a relay restarts.

Global Reach and Latency Optimization

OpenAI utilizes a "Global Relay" fleet of geographically distributed ingress points to shorten the first client-to-OpenAI hop. This reduces round-trip time (RTT), jitter, and packet loss before traffic enters the OpenAI backbone.

• Geo-Steered Signaling: Cloudflare geo and proximity steering ensure the initial HTTP or WebSocket request reaches a nearby transceiver cluster.
• Integrated Routing: The SDP (Session Description Protocol) answer provides the Global Relay address, while the ufrag ensures the relay can route the media to the correct cluster and transceiver.

Technical Implementation and Performance

The relay service is written in Go and optimized for high throughput without requiring a kernel-bypass framework. Key performance optimizations include:

• SO_REUSEPORT: This Linux socket option allows multiple relay workers to bind to the same UDP port, letting the kernel distribute incoming packets across workers to avoid read-loop bottlenecks.
• runtime.LockOSThread: This pins UDP-reading goroutines to specific OS threads, improving cache locality and reducing context switching by keeping packets from the same flow on the same CPU core.
• Memory Management: The use of pre-allocated buffers and minimal copying reduces allocation overhead and garbage collection pauses.

Summary of Architectural Gains

By moving complexity into a thin routing layer rather than the backend services or the client, OpenAI achieved a scalable WebRTC deployment that preserves standard protocol semantics. This ensures that browsers and mobile apps remain interoperable while allowing the inference backend to scale like ordinary services rather than acting as WebRTC peers.