OpenAI o3 and o4-mini Visual Reasoning Release

OpenAI o3 and o4-mini introduce the ability for models to "think with images," meaning they can now perform image manipulation—such as cropping, zooming, and rotating—within their internal chain-of-thought to solve complex visual problems. This marks a transition from models that simply see images to models that can actively reason about them using a multimodal agentic approach.

Native Image Manipulation in Chain-of-Thought

OpenAI o3 and o4-mini extend the long-form internal reasoning capabilities of the o-series by integrating visual tools natively. Unlike previous multimodal models, these models do not rely on separate specialized models for image processing; instead, they can automatically apply the following techniques to user-uploaded images during their reasoning process:

• Zooming and Cropping: Focusing on specific areas of an image to extract finer details or read small text.
• Rotation and Flipping: Correcting the orientation of images (e.g., rotating upside-down text) to make them readable.
• Enhancement: Applying simple image processing techniques to improve clarity.

This capability allows the models to handle imperfect photos, such as those with poor positioning, upside-down text, or multiple problems within a single frame, by iteratively refining their view of the data before arriving at a final answer.

Multimodal Agentic Capabilities

By blending visual reasoning with other tools, o3 and o4-mini provide a multimodal agentic experience. The models can combine their internal image manipulation with external tools such as web search and Python data analysis to solve high-complexity tasks.

Examples of these capabilities include:

• Document Analysis: Reading handwriting from a notebook by rotating and zooming into the text.
• Complex Problem Solving: Solving a maze by using Python to analyze alpha channels and pixel values to plot a valid path.
• Technical Troubleshooting: Performing root-cause analysis on screenshots of build errors.
• Educational Support: Providing step-by-step explanations for economics problem sets uploaded via photo.

Benchmark Performance and Scaling

The introduction of "thinking with images" creates a new axis for test-time compute scaling, blending visual and textual reasoning. When evaluated at high "reasoning effort" settings, o3 and o4-mini significantly outperform previous multimodal models across all tested tasks.

Key performance milestones include:

• State-of-the-Art (SOTA) Results: The models achieved SOTA performance in STEM question-answering (MMMU, MathVista), chart reading and reasoning (CharXiv), perception primitives (VLMs are Blind), and visual search (V*).
• Visual Search Mastery: On the V* benchmark, the visual reasoning approach achieved 95.7% accuracy, largely solving the benchmark.

Current Limitations

Despite the advancements, OpenAI identifies three primary limitations in the current visual reasoning implementation:

1. Inefficient Reasoning Chains: Models may perform redundant tool calls or unnecessary image manipulations, leading to excessively long chains of thought.
2. Perception Errors: Basic perception mistakes can still occur; a model may correctly use a tool to zoom in but still misinterpret the resulting visual data.
3. Reliability Issues: The models may employ different visual reasoning paths across multiple attempts at the same problem, leading to inconsistent results.

OpenAI is currently refining these models to be more concise and reliable in their multimodal reasoning processes.