AI Reasoning and the Illusion of Thinking: Are Large Reasoning Models Right for the Wrong Reasons?

The Paradox of AI Reasoning: High Performance vs. Low Faithfulness

Large Reasoning Models (LRMs) are currently producing results that challenge the boundaries of mathematical research, including solving open research problems and winning gold medals at the International Mathematical Olympiad. However, a growing body of evidence suggests a fundamental disconnect between the models' final answers and the "chains of thought" (CoT) they generate to reach them. The central tension in current AI research is whether these models are performing genuine logical reasoning or are simply "right for the wrong reasons."

The "Illusion of Thinking": Unfaithful Reasoning Traces

While LRMs are trained to produce intermediate steps (reasoning traces) to improve accuracy, research indicates these traces are often not faithful representations of the model's internal computation.

Non-Causal Thinking Tokens

Research from Northeastern University and UC Berkeley found that between 30% and 60% of thinking steps in frontier open-source LRMs had minimal causal impact on the final answer. In many cases, removing a significant portion of these steps did not degrade performance.

Meaningless Filler and "Mumblings"

Studies have demonstrated that the linguistic content of reasoning traces can be largely irrelevant:

Filler Tokens: Research from NYU showed that strings of dots (meaningless filler tokens) could function effectively in place of human-readable chains of thought.
Trace Replacement: Subbarao Kambhampati's lab showed that replacing correct reasoning traces with incorrect or irrelevant ones did not necessarily degrade performance on formal reasoning tasks.
Faithfulness Gap: Models trained exclusively on correct trace data still occasionally generate invalid records of their reasoning even when the final solution is correct.

Theoretical Frameworks: Approximate Retrieval vs. Genuine Reasoning

To explain how models can be accurate without faithful reasoning, researchers have proposed several competing hypotheses.

The Approximate Retrieval Hypothesis

Subbarao Kambhampati suggests that LRMs do not perform step-by-step logical reasoning. Instead, they perform "approximate retrieval"—a process between pattern matching and reasoning. In this view, "thinking tokens" serve as a mechanism to load the model's context window, making it more likely to predict "reasoning-shaped" strings of text from its training corpus. This is akin to "mumbling to oneself" to jog a memory rather than executing a formal algorithm.

The Emulated Recursion Hypothesis

Some technical perspectives suggest that since Transformers have a fixed depth and lack native recursion, reasoning traces are a way to emulate deeper recursion. By generating tokens, the model can refine concepts over a longer sequence than the network's layers would allow, effectively using the KV cache to maintain state across multiple iterations of refinement.

The Verifiable Domain Advantage

The success of LRMs in coding and mathematics is attributed to the nature of these "verifiable domains." Because code and proofs have binary outcomes (they work or they don't), they provide strong training signals. Models may not be learning general reasoning algorithms but are instead absorbing enough examples of what correct steps look like to predictively mimic them until a verifiable result is reached.

Industry Perspectives and the "Wishful Mnemonic" Problem

There is a sharp divide between academic skeptics and industry practitioners regarding the interpretation of these results.

The Industry View

Practitioners, such as those at OpenAI, argue that the utility of the model outweighs the need for a mechanical explanation. They contend that if a model can solve a complex mathematical proof, it is reasoning, regardless of whether the intermediate tokens are perfectly faithful. Some argue that earlier critiques of AI reasoning were based on obsolete models and that newer iterations (e.g., GPT-5.5) have overcome these issues.

The "Wishful Mnemonic" Critique

Critics argue that using terms like "reasoning," "thinking," and "understanding" are "wishful mnemonics"—labels that beg the question by assuming the very ability they are trying to prove. This anthropomorphism can lead researchers to believe a model is performing a cognitive process simply because the output looks like a human's description of that process.

Synthesis of Community Insights

Discussion among technical observers highlights several key counterpoints and analogies:

The "Clever Hans" Analogy: Some compare LRMs to Clever Hans, the horse that appeared to do math but was actually reading cues from its handler. In this context, LRMs may be optimizing for the "reward signal" of the human evaluator rather than the logical truth of the problem.
The Semantics Debate: Some argue that the distinction between "true reasoning" and "approximate retrieval" is a matter of semantics. If a system consistently produces correct results for complex problems, the internal mechanism—whether it be matrix multiplication or logical deduction—is secondary to the functionality.
The X-Ray Analogy: A warning that using "magic" technology without understanding its internal mechanisms is like the early use of X-rays: it works brilliantly until the hidden damage (or failure modes) becomes apparent.