OpenAI Realtime API Release

OpenAI has introduced the Realtime API in public beta, allowing paid developers to integrate low-latency, multimodal speech-to-speech conversations into their applications. This API eliminates the need to stitch together multiple models for transcription, reasoning, and speech synthesis, enabling natural conversational experiences with a single API call.

Technical Architecture and Workflow

The Realtime API replaces the traditional three-step pipeline—Automatic Speech Recognition (ASR) via models like Whisper, a text-based LLM for reasoning, and a Text-to-Speech (TTS) model—with a direct streaming approach. By streaming audio inputs and outputs directly, the API reduces latency and preserves emotional nuance, emphasis, and accents that are typically lost in text-based transcription.

Key technical features include:

  • Persistent WebSocket Connection: The API utilizes WebSockets to exchange messages with GPT-4o in real-time.
  • Automatic Interruption Handling: The system can handle user interruptions automatically, mirroring the behavior of ChatGPT's Advanced Voice Mode.
  • Function Calling: The API supports function calling, allowing voice assistants to trigger external actions or retrieve real-time context, such as placing orders or accessing customer data.

Model Availability and API Options

OpenAI provides two distinct paths for audio integration depending on the latency requirements of the application:

  1. Realtime API: Powered by the gpt-4o-realtime-preview model, this is designed for high-speed, natural conversations.
  2. Chat Completions API: Powered by the gpt-4o-audio-preview model, this allows developers to pass text or audio inputs and receive responses in text, audio, or both. This is intended for use cases that do not require the extreme low-latency benefits of the Realtime API.

Pricing and Tokenization

The Realtime API and the Chat Completions audio capabilities use a unified pricing structure based on text and audio tokens.

Input Type Price per 1M Tokens Estimated Cost per Minute
Text Input $5.00 N/A
Text Output $20.00 N/A
Audio Input $100.00 ~$0.06
Audio Output $200.00 ~$0.24

Additionally, cached pricing is available for text and audio inputs, reducing costs to $2.50 per 1M cached text input tokens and $20 per 1M cached audio input tokens.

Safety, Privacy, and Compliance

The Realtime API is built on the same GPT-4o version used in ChatGPT's Advanced Voice Mode and incorporates the same audio safety infrastructure. Safety measures include:

  • Automated Monitoring: Continuous monitoring and human review of flagged inputs and outputs to mitigate abuse.
  • ** uma Preparedness Framework**: Evaluations conducted according to OpenAI's Preparedness Framework and detailed in the GPT-4o System Card.
  • Red Teaming: External red teaming confirmed that the API does not introduce high-risk gaps beyond existing mitigations.
  • Privacy Commitments: Under Enterprise privacy commitments, OpenAI does not train models on inputs or outputs from this service without explicit permission.

Integration and Ecosystem

To facilitate deployment, OpenAI has collaborated with several infrastructure partners:

  • LiveKit and Agora: Provided client libraries for audio components such as sound isolation, echo cancellation, and reconnection.
  • Twilio: Integrated the Realtime API with Twilio's Voice APIs to connect AI agents to customers via traditional voice calls.

Future Roadmap

OpenAI plans to expand the Realtime API's capabilities as it moves toward general availability, including:

  • Additional Modalities: Future support for vision and video.
  • Increased Rate Limits: Scaling simultaneous session limits beyond the current Tier 5 cap of 100 sessions.
  • Official SDK Support: Integration into the official OpenAI Python and Node.js SDKs.
  • Expanded Model Support: Integration of GPT-4o mini into upcoming versions of the model.

Sources