OpenAI o1 and Developer Tooling Updates December 2024
OpenAI has introduced the production-ready o1 reasoning model to its API, alongside updates to the Realtime API, a new Preference Fine-Tuning method, and beta SDKs for Go and Java. These updates focus on increasing the reasoning capabilities of agentic applications, reducing the cost of real-time audio interactions, and providing more granular control over model behavior.
OpenAI o1 API Integration
OpenAI o1 is a reasoning model designed for complex, multi-step tasks, now available to developers on usage tier 5. This model succeeds o1-preview and is optimized for production environments with several new technical capabilities.
Key Technical Features
- Function Calling and Structured Outputs: o1 can now connect to external APIs and data sources and reliably generate responses adhering to custom JSON Schemas.
- Vision Capabilities: The model can reason over visual inputs, enabling applications in coding, manufacturing, and science.
- Developer Messages: Developers can now provide specific behavioral guidance, such as tone and style, via developer messages.
- Reasoning Control: A new
reasoning_effortAPI parameter allows developers to control the duration of the model's thinking process before it generates an answer. - Efficiency: The o1 model uses an average of 60% fewer reasoning tokens than o1-preview for the same request.
Performance Benchmarks for o1-2024-12-17
The latest snapshot (o1-2024-12-17) demonstrates state-of-the-art results across several domains compared to o1-preview:
| Category | Eval | o1-2024-12-17 | o1-preview |
|---|---|---|---|
| General | GPQA diamond | 75.7 | 73.3 |
| General | MMLU (pass @1) | 91.8 | 90.8 |
| Coding | SWE-bench Verified | 48.9 | 41.3 |
| Coding | LiveBench (Coding) | 76.6 | 52.3 |
| Math | MATH (pass @1) | 96.4 | 85.5 |
| Math | AIME 2024 (pass @1) | 79.2 | 42.0 |
| Math | MGSM (pass @1) | 89.3 | 90.8 |
| Vision | MMMU (pass @1) | 77.3 | — |
| Vision | MathVista (pass @1) | 71.0 | — |
| Factuality | SimpleQA | 42.6 | 42.4 |
| Agents | TAU-bench (retail) | 73.5 | — |
| Agents | TAU-bench (airline) | 54.2 | — |
OpenAI reports that o1-2024-12-17 significantly outperforms gpt-4o in Structured Outputs and function calling tests.
Realtime API Enhancements
Updates to the Realtime API focus on improving accessibility for voice-based applications and reducing operational costs.
WebRTC Integration
OpenAI has added support for WebRTC, an open standard that simplifies the scaling of real-time voice products across browsers, mobile clients, and IoT devices. This integration handles congestion control, noise suppression, audio streaming, and encoding to ensure responsive interactions despite variable network quality.
Pricing and Model Updates
Two new snapshots have been released for the Realtime API beta:
- gpt-4o-realtime-preview-2024-12-17: Features improved voice quality and more reliable input for dictated numbers. Audio token prices have been reduced by 60% ($40/1M input, $80/1M output). Cached audio input costs have dropped by 87.5% to $2.50/1M input tokens.
- gpt-4o-mini-realtime-preview-2024-12-17: Brings the cost-efficiency of the mini model to the Realtime API. Audio pricing is set at $10/1M input and $20/1M output tokens; text tokens are $0.60/1M input and $2.40/1M output tokens.
Preference Fine-Tuning
OpenAI has introduced Preference Fine-Tuning (PFT), a customization technique using Direct Preference Optimization (DPO) to tailor models based on pairwise comparisons of responses.
PFT vs. Supervised Fine-Tuning (SFT)
Unlike SFT, which replicates labeled outputs, PFT optimizes the model to favor preferred behaviors over non-preferred ones.
| Feature | Supervised Fine-Tuning (SFT) | Preference Fine-Tuning (PFT) |
|---|---|---|
| Objective | Replicate labeled outputs for correctness | Reinforce preferred responses and reduce unpreferred ones |
| Training Data | Exact input/output pairs | Pairs of preferred and non-preferred outputs |
| Use Cases | Strict correctness, custom code formats | Subjective tasks like creative writing or summarization |
Real-World Application
Rogo AI utilized PFT to improve a financial AI assistant's ability to handle out-of-distribution query expansion. By using their Rogo-Golden benchmark, they increased accuracy from 75% (base model) to over 80%.
Preference Fine-Tuning is currently rolling out for gpt-4o-2024-08-06 and will soon be available for gpt-4o-mini-2024-07-18.
New SDKs for Go and Java
OpenAI has released beta SDKs for Go and Java to expand developer accessibility:
- Go SDK: Provides a dedicated library for API integration.
- Java SDK: Offers typed request and response objects and utilities to manage API requests, targeting enterprise software development environments.