OpenAI Next-Generation Audio Models API Release

OpenAI has launched a new suite of audio models in its API to enable the development of more intelligent and customizable voice agents. These models introduce state-of-the-art speech-to-text accuracy and the first steerable text-to-speech capabilities, allowing developers to control how a model speaks based on specific instructions.

Enhanced Speech-to-Text Capabilities

OpenAI has introduced gpt-4o-transcribe and gpt-4o-mini-transcribe, which outperform previous Whisper models in accuracy and reliability. These models are specifically designed to handle challenging audio environments, including varying speech speeds, noisy backgrounds, and diverse accents.

Key improvements include:

  • Reduced Word Error Rate (WER): The new models achieve lower WER across multiple benchmarks, including the multilingual FLEURS benchmark covering over 100 languages.
  • Benchmark Performance: The models consistently outperform Whisper v2 and Whisper v3 across all language evaluations.
  • Use Case Optimization: Increased transcription reliability makes these models particularly effective for meeting note transcription and customer call centers.

Steerable Text-to-Speech

The new gpt-4o-mini-tts model introduces steerability, allowing developers to instruct the model on the delivery and tone of the speech. For the first time, developers can provide instructions such as "talk like a sympathetic customer service agent" to customize the emotional resonance and style of the voice.

This capability enables a variety of tailored applications, including expressive narration for storytelling and empathetic customer service interactions. These models are currently limited to artificial, preset voices to ensure they match synthetic presets.

Technical Innovations

The next-generation audio models are built on GPT-4o and GPT-4o-mini architectures and utilize several key technical advancements:

Pretraining and Datasets

  • Audio-Centric Pretraining: The models were extensively pretrained on specialized audio-centric datasets to optimize performance and provide deeper insight into speech nuances.

Distillation Methodologies

  • Advanced Distillation: OpenAI used enhanced distillation techniques and self-play methodologies to transfer knowledge from larger audio models to smaller, more efficient versions. This ensures smaller models maintain high conversational quality and responsiveness by replicating realistic user-assistant interactions.

Reinforcement Learning

  • RL-Heavy Paradigm: For speech-to-text models, a reinforcement learning-heavy approach was integrated to push transcription accuracy to state-of-the-art levels, specifically reducing hallucinations and improving precision in complex recognition scenarios.

API Availability and Integration

These models are available to all developers via the speech-to-text and text-to-speech APIs. To simplify the development of voice agents, OpenAI has released an integration with the Agents SDK. For developers requiring low-latency speech-to-speech experiences, OpenAI recommends the speech-to-speech models available in the Realtime API.

Future Roadmap

OpenAI plans to continue improving audio model intelligence and accuracy. Future goals include exploring ways for developers to bring their own custom voices while maintaining safety standards. Additionally, OpenAI is investing in other modalities, including video, to support the creation of multimodal agentic experiences.

Sources