OpenAI o1-mini release notes
OpenAI has launched o1-mini, a specialized reasoning model designed for high performance in STEM fields at a lower cost and higher speed than the full o1 series. This model is intended for applications that require complex reasoning without the need for broad, general-world knowledge.
Cost and Availability
o1-mini is available to Tier 5 API users at a cost that is 80% cheaper than o1-preview. For ChatGPT Plus, Team, Enterprise, and Edu users, o1-mini serves as an alternative to o1-preview, providing higher rate limits and lower latency.
STEM Reasoning Performance
o1-mini is a smaller model optimized for STEM reasoning during pretraining and trained using the same high-compute reinforcement learning (RL) pipeline as the full o1 model. While it lacks the broad world knowledge of higher-capacity models, it achieves comparable reasoning performance in specific technical domains.
Mathematics and Coding
In technical benchmarks, o1-mini demonstrates competitive performance against the full o1 model and outperforms o1-preview in several areas:
- Mathematics: On the high school AIME math competition, o1-mini scored 70.0%, placing it in approximately the top 500 US high-school students. This outperforms o1-preview (44.6%) and is competitive with o1 (74.4%).
- Coding: On Codeforces, o1-mini achieved an Elo of 1650, placing it in the 86th percentile of competing programmers. This is higher than o1-preview (1258) and competitive with o1 (1673). The model also performs well on the HumanEval benchmark and high-school level cybersecurity capture the flag (CTF) challenges.
General STEM and Human Preference
o1-mini outperforms GPT-4o on academic benchmarks requiring reasoning, such as MATH-500 and GPQA (science). However, it lags behind o1-preview on GPQA and performs worse than GPT-4o on MMLU due to its limited non-STEM factual knowledge.
Human preference evaluations indicate that o1-mini is preferred over GPT-4o for reasoning-heavy domains, though it is not preferred for language-focused tasks.
Model Speed and Latency
o1-mini provides significantly faster response times than its larger counterparts. In a word reasoning test, o1-mini reached the correct answer approximately 3-5x faster than o1-preview, while GPT-4o failed to answer the question correctly.
Safety and Robustness
o1-mini utilizes the same alignment and safety techniques as o1-preview. Internal evaluations using the StrongREJECT dataset show that o1-mini has 59% higher jailbreak robustness compared to GPT-4o.
| Metric | GPT-4o | o1-mini |
|---|---|---|
| % Safe completions refusal on harmful prompts (standard) | 0.99 | 0.99 |
| % Safe completions on harmful prompts (Challenging: jailbreaks & edge cases) | 0.714 | 0.932 |
| % Compliance on benign edge cases | 0.91 | 0.923 |
| Goodness@0.1 StrongREJECT jailbreak eval | 0.22 | 0.83 |
| Human sourced jailbreak eval | 0.77 | 0.95 |
Limitations
Because o1-mini is specialized for STEM, its factual knowledge regarding non-STEM topics—including biographies, dates, and general trivia—is comparable to smaller models like GPT-4o mini. OpenAI intends to address these limitations in future versions and explore extensions into other modalities and specialties.
Sources
- OriginalOpenAI o1-mini