OpenAI gpt-realtime and Realtime API General Availability

OpenAI has announced the general availability of the Realtime API and the release of gpt-realtime, a new speech-to-speech model designed for production-ready voice agents. This update introduces native support for image inputs, remote Model Context Protocol (MCP) servers, and the Session Initiation Protocol (SIP) for phone network integration, significantly expanding the utility of voice-based AI agents.

The gpt-realtime Model

gpt-realtime is a native speech-to-speech model that processes and generates audio directly, eliminating the need for separate speech-to-text and text-to-speech pipelines. This architecture reduces latency and preserves the nuance of human speech.

Audio Quality and Expressiveness

The model is trained to produce more natural intonation, emotion, and pace. It can follow fine-grained instructions regarding style, such as speaking "empathetically in a French accent" or "quickly and professionally." OpenAI has introduced two new exclusive voices, Cedar and Marin, while updating the existing eight voices to leverage these quality improvements.

Intelligence and Reasoning

gpt-realtime demonstrates improved comprehension of native audio and non-verbal cues, such as laughter. It can switch languages mid-sentence and more accurately detect alphanumeric sequences (e.g., phone numbers or VINs) in Spanish, Chinese, Japanese, and French.

In terms of reasoning, the model achieved 82.8% accuracy on the Big Bench Audio benchmark, compared to 65.6% for the December 2024 model.

Instruction Following and Function Calling

Improvements in instruction following allow the model to adhere more strictly to developer prompts, such as reading disclaimer scripts word-for-word. On the MultiChallenge audio benchmark, gpt-realtime scored 30.5%, up from 20.6% in the previous model.

Function calling capabilities have been enhanced across three dimensions: the relevance of the function called, the timing of the call, and the accuracy of the arguments. On the ComplexFuncBench audio eval, gpt-realtime scored 66.5%, compared to 49.7% for the previous model. Additionally, the model now natively supports asynchronous function calling, allowing conversations to remain fluid while waiting for long-running tool results.

Realtime API Feature Updates

Beyond the model update, the Realtime API introduces several production-oriented features:

  • Remote MCP Server Support: Developers can enable Model Context Protocol (MCP) support by providing a remote MCP server URL in the session configuration, allowing the API to handle tool calls automatically without manual integration.
  • Image Input: The model can now process images, photos, and screenshots alongside audio or text. This allows the agent to ground conversations in visual context, enabling users to ask questions about what the model "sees" in a shared image.
  • SIP Support: Direct support for the Session Initiation Protocol (SIP) allows developers to connect voice agents to the public phone network, PBX systems, and desk phones.
  • Reusable Prompts: Developers can now save and reuse prompts—including tools, variables, and example messages—across different Realtime API sessions.

Safety, Privacy, and Pricing

To prevent misuse, OpenAI employs active classifiers to monitor sessions and can halt conversations that violate harmful content guidelines. Developers are encouraged to use the Agents SDK for additional guardrails. Usage policies require that developers disclose to users when they are interacting with AI and prohibit the use of the services for spam or deception.

Pricing and Availability

gpt-realtime is available to all developers immediately. Pricing has been reduced by 20% compared to gpt-4o-realtime-preview:

  • Audio Input: $32 per 1 million tokens ($0.40 for cached input tokens).
  • Audio Output: $64 per 1 million tokens.

Developers now have fine-grained control over conversation context, allowing them to set token limits and truncate turns to manage costs in long sessions.

Sources