OpenAI o3-mini release notes / what's new

OpenAI o3-mini is a cost-efficient reasoning model optimized for STEM (science, math, and coding) that delivers performance comparable to the larger OpenAI o1 model while maintaining the low cost and reduced latency of o1-mini. It is available now in ChatGPT and via the API for select developers.

STEM Reasoning and Performance

OpenAI o3-mini is designed specifically for technical domains requiring high precision and speed. With "medium" reasoning effort, it matches the performance of OpenAI o1 in math, coding, and science.

Technical Benchmarks

  • Mathematics: In AIME 2024 evaluations, o3-mini with medium effort matches o1, while high reasoning effort allows it to outperform both o1 and o1-mini.
  • PhD-level Science: On the GPQA Diamond benchmark (biology, chemistry, and physics), o3-mini with low effort already exceeds o1-mini, and with high effort, it reaches parity with o1.
  • Research-level Math: On FrontierMath, o3-mini with high reasoning effort and a Python tool solves over 32% of problems on the first attempt, including more than 28% of the challenging T3 problems.
  • Competitive Coding: On Codeforces, o3-mini outperforms o1-mini across all reasoning effort levels and matches o1's performance at medium effort.
  • Software Engineering: o3-mini is OpenAI's highest performing released model on SWE-bench Verified (n=477). Using an internal tools scaffold, it achieves a 61% success rate.
  • LiveBench Coding: o3-mini surpasses o1-high even at medium reasoning effort.

Human Preference and Accuracy

External expert testers preferred o3-mini's responses to o1-mini 56% of the time and reported a 39% reduction in major errors on difficult real-world questions.

Model Speed and Efficiency

OpenAI o3-mini provides significant latency improvements over its predecessor, o1-mini. In A/B testing, o3-mini delivered responses 24% faster than o1-mini, with an average response time of 7.7 seconds compared to 10.16 seconds. Additionally, o3-mini has an average time to first token that is 2,500ms faster than o1-mini.

Developer Features and API Integration

OpenAI o3-mini is the first small reasoning model to support several production-ready developer features:

  • Function Calling: Support for integrating the model with external tools.
  • Structured Outputs: Ensuring the model adheres to specific JSON schemas.
  • Developer Messages: Enhanced prompting capabilities for system-level instructions.
  • Reasoning Effort: Developers can select between low, medium, and high reasoning effort to balance speed and intelligence for specific use cases.
  • Streaming: Support for streaming responses, consistent with o1-mini and o1-preview.

Note: o3-mini does not support vision capabilities; visual reasoning tasks should continue to be handled by OpenAI o1.

Availability and Access

  • ChatGPT Plus, Team, and Pro: Available immediately. Pro users have unlimited access to both o3-mini and o3-mini-high.
  • Free Users: For the first time, a reasoning model is available to free users in ChatGPT via the 'Reason' option in the message composer.
  • Enterprise: Access is scheduled for February.
  • API: Rolling out to developers in usage tiers 3-5 via Chat Completions, Assistants, and Batch APIs.
  • Rate Limits: Rate limits for Plus and Team users have been tripled from 50 messages per day (o1-mini) to 150 messages per day (o3-mini).

Safety and Alignment

OpenAI utilized "deliberative alignment" to train o3-mini, a process where the model reasons about human-written safety specifications before generating a response. This approach allows o3-mini to significantly surpass GPT-4o on challenging safety and jailbreak evaluations. The model underwent external red-teaming and safety evaluations consistent with the preparedness approach used for OpenAI o1.

Sources