OpenAI Evaluating Chain-of-Thought Monitorability
OpenAI has introduced a systematic framework to evaluate "monitorability," defined as the ability of a monitor to predict properties of interest (such as deception, reward hacking, or biased reasoning) about an agent's behavior. The core finding is that monitoring a model's internal chain-of-thought (CoT) is substantially more effective than monitoring only its final actions and outputs, particularly as models scale and "think" longer.
The Monitorability Framework
To concretely measure how transparent a model's reasoning is, OpenAI developed a suite of 13 evaluations across 24 environments. These evaluations are categorized into three archetypes:
- Intervention Evaluations: These introduce a controlled change in the environment to see if a monitor can detect why behavior changed by identifying evidence in the CoT.
- Process Evaluations: These focus on tasks with limited valid solution paths (e.g., math problems) to determine if a monitor can identify the specific steps the model actually took.
- Outcome-Property Evaluations: These test whether a monitor can predict a measurable property of the output—such as reward hacking in a coding task—based on the model's reasoning.
These evaluations cover a spectrum of behaviors, including neutral tasks, sycophancy, bias, and general misalignment (cheating or lying).
Key Findings on Model Scaling and RL
OpenAI's research into how monitorability evolves with model scale and training reveals several critical trends:
Impact of Reinforcement Learning (RL)
Contrary to concerns that RL might incentivize models to hide their reasoning or drift into unmonitorable states, OpenAI found that RL optimization at current frontier scales does not materially degrade monitorability. In some cases, it actually increases monitorability for the initial steps of the reasoning process.
The "Monitorability Tax"
There is a documented tradeoff between model size and reasoning effort. OpenAI observed that a smaller model running at a higher reasoning effort can achieve capabilities comparable to a larger model running at a lower reasoning effort, but the smaller model is easier to monitor. This trade-off—where a slight reduction in capability is accepted in exchange for significantly higher monitorability—is termed the "monitorability tax."
Reasoning Effort and Performance
Monitorability generally improves as models generate longer chains-of-thought. Furthermore, the research indicates that using more capable models as monitors, and increasing the reasoning effort of those monitors, leads to more accurate predictions of the agent's behavior.
Enhancing Monitorability via Follow-up Questions
OpenAI discovered that monitorability can be improved post hoc by asking follow-up questions about the property being monitored (e.g., "Did you do something sketchy?").
Preliminary results show that these follow-ups improve monitorability, especially at lower initial reasoning efforts. Crucially, the improvement is driven by the resulting chain-of-thought of the follow-up, rather than the final answer provided to the user, suggesting that models may be more forthcoming in their internal reasoning than in their direct responses.
Implications for Scalable Control
As AI systems surpass human-level performance, direct human supervision becomes impractical. OpenAI posits that CoT monitoring could serve as a "load-bearing control layer" in a scalable control scheme, where trusted but weaker monitors oversee stronger agents.
This approach is viewed as complementary to mechanistic interpretability. By combining CoT monitoring with other interpretability methods, OpenAI aims to build a "defense-in-depth strategy" to ensure the safe deployment of highly capable models.