Neural Networks Are Already Clean: Cammarata's Case That Superposition Was a Misreading

Nick Cammarata's essay "Neural Networks Are Already Clean," published at already-clean.goodfire.com and discussed on Hacker News (801 points, 325 comments), argues that the field's core justification for abandoning direct study of neurons — superposition — was a misreading of evidence, and that modern neural networks are legible enough to study directly with ordinary science.

The essay's central claim is that a network's own mechanisms are its legible building blocks, and that mechanistic interpretability may be far more tractable than the field feared. Cammarata writes that "what looked like arbitrary packing was mostly a motif I'll call recursive generalization, which is much less concerning."

The cat-and-car neuron is a V-shape neuron

The most famous example of a polysemantic neuron is unit 4e:55 in InceptionV1, which appeared to respond to both cat faces and the fronts of cars. Cammarata, who co-authored the 2020 Distill paper Zoom In that popularized the example, now says the neuron is a V-shape detector.

Looking at the neuron's circuit connections, he found V-shaped flames, V-shaped water jets from a fireboat, and V-shaped fox faces. Its negative pole responds to an O — round where the V is pointed. Downstream readers include a flame neuron, a coyote-face neuron, and a martini-glass neuron.

Cammarata attributes the original misreading to two compounding errors. First, the field did not know about generalized pose neurons, of which InceptionV1 has more than a hundred. Second, feature visualization was reward-hacked in deeper layers: gradient descent takes the easiest path, which in InceptionV1 often means dogs, because the training data contains many dogs and many dog neurons. "If you even lightly bias it toward the natural paths the top images take, most of them are cars, and some are both dogs and cars, because of shared pose."

The same error affected the canonical superposition circuit Cammarata found in 2019, which appeared to show a car feature spreading into a set of unrelated dog neurons. He now says cars and dogs were brought together by shared pose, and that nothing directly reads out the combination. "It wasn't hiding anything; it was ordinary generalization and composition. To my knowledge, no one has published a second circuit with the right shape."

InceptionV1's last layer was a learning failure, not superposition

The essay's second major claim is that InceptionV1's supposedly uninterpretable late layers were an architectural problem. Cammarata recalls that when the field first studied the model, almost none of the neurons in its last layer were easily interpretable. One estimate he cites said "To the best of my knowledge, there are essentially no interpretable neurons on InceptionV1's final layer."

With better interpretability tools, he now puts the figure at roughly 50% on the same network. Retraining the same architecture with more reasonable modern regularization raises it to about 70%. A ConvNeXt of the same size and trained on the same dataset reaches about 99%.

The architectural changes that fixed this were not designed for interpretability. Residual connections and LayerNorm were introduced to help deep networks learn. Cammarata argues they also made deep layers conceptually clean. "With these ordinary improvements, and no change to model size or training data, 99-100% of neurons in that last layer are easily interpretable and the model is broadly legible the whole way through."

He trained ConvNeXt models spanning roughly 300× in size — 92k, 7.06M, and 28.6M parameters — and reports that the networks are broadly legible at every scale. Smaller models have coarser neurons, not fractured ones. "With less room, the network doesn't fold unrelated things together; its neurons just cover coarser-grained concepts, like one car neuron covering many kinds of car."

Recursive generalization replaces superposition

Cammarata's proposed alternative to superposition is a motif he calls recursive generalization: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. A language-model neuron that detects "iy" grows to detect "diy," and then "do it yourself," which contains no "iy" at all.

He says the idea first appeared in a forgotten subsection of the 2021 Multimodal Neurons paper, where it was called "conjoined features." At the time, he had not watched neurons grow over training. Now he says it holds up everywhere he has looked, including in the original InceptionV1, whose least legible neurons "turn out to be overgrowing locally into reasonable, adjacent territory."

The distinction matters because the two explanations make opposite predictions. Generalization groups similar things together, so the grouping is a clue to how the network thinks. Superposition groups unrelated things together, so the grouping teaches almost nothing. "Superposition is the one the field bet on and, until recently, so did I. But I now think recursive generalization was basically right."

How interpretable are modern models?

Cammarata reports preliminary audits of Qwen3-32B in which roughly 90% of neuron poles have one main job, often with small related offshoots; about 7% have two related jobs; and only about 2-4% are not yet understood. He notes that the exact number depends on where thresholds are set for weak activations, depth, and what counts as "one thing."

He contrasts this with earlier results. Anthropic's 2023 Towards Monosemanticity paper, studying a small one-layer model, gave the median neuron a 0 out of 14 on its rubric, meaning the rater could not form any hypothesis. The OpenAI automated interpretability project Cammarata started in 2023 found about 1.7% of neurons interpretable by its most generous measure.

For GPT-2 XL, the model used in that original project, he reports that fewer than 2% of neurons came out clean at the time. Going back with modern tools and making 500 neuron papers with almost no iteration, about 70% now look like roughly one thing. About 5% are pristine.

He also reports that the same results hold in models up to trillions of parameters, including Kimi K3, which the companion paper will cover in more detail. The essay includes what it describes as the first published attribution graph of a multi-trillion-parameter model.

The crushed-layer experiments

To test whether networks switch to superposition under compression, Cammarata retrained InceptionV1 with layer 3b cut from 480 neurons to 10, with no bypass around the layer. The model reached 58.35% top-1 and 81.14% top-5 accuracy on 50,000 held-out ImageNet images.

If superposition were the fallback, the layer should have shown overlapping mixtures of curves, colors, and textures. Instead, the neurons stayed interpretable but coarser: one "curves & loops" neuron instead of ten curve detectors, two circle neurons instead of several circle types.

He then trained a TopK sparse autoencoder with 640 features on the squeezed layer. The SAE explained 82% of the variance of held-out positions, and with it spliced in, top-1 on 10,000 validation images fell from 58.8% to 43.0%. The SAE features that best matched the original curve detectors were broad and bunched together, not clean curves. "We don't see anything like clean curve neurons."

A parallel experiment crushed the last layer of the 7.06M-parameter ConvNeXt from 1,024 neurons to 13 and trained from scratch. The result was the same pattern of broadening.

Weak activations are mostly generalization

Cammarata argues that weak activations are not where hidden features hide. His reasoning is that weak activations are vastly denser than strong ones, so a packed feature would collide with orders of magnitude more things for little upside.

He reports that removing the weakest 75% of activations in the medium ConvNeXt drops accuracy from 72% to 71%, with 1 in 10 answers changing. Removing the strongest 7% drops accuracy from 72% to 4%, with 96 in 100 answers changing. "Critical common things can't be in the weak activations, since we can remove them."

He also argues against percentile-based thinking. "The 99th percentile sounds high, but 1 token in 100 is about a quarter of a page, and a neuron for a niche feeling or social context — which are all over language models — shouldn't come up four times a page."

Language-model examples

Several concrete language-model neurons are used to illustrate the argument. A neuron that standard top-k automated interpretability labels as a "polysemantic comma neuron" turns out to track Colorado: phone numbers, slang, and connections to other Colorado neurons. Steering it made the model want to go hiking. The commas are where the model stores a running summary of the Colorado context.

A neuron that fires on "pr" also fires on Base64 URL strings and Bengali text. Decoding the URLs produces "pr" words; transliterating the Bengali produces "pr".

A geology neuron that generalizes into Cesium, a JavaScript library, is presented as a recursive generalization offshoot. A Miyazaki neuron fires on the name in every script and sense the authors tried, and feeds later associations like "spirited away."

The SwiGLU double-negative regime

Cammarata reports finding a double-negative regime in SwiGLU neurons that, as far as he knows, no one had noticed. Modern neurons have both a gate and a value; when both are negative, the product is positive, so a neuron can fire for two quite different reasons.

In one example, a neuron whose top-k dashboard shows "References to the book of Colossians" turns out to have its gate closed on every one of those activations. The neuron's actual job is the token that completes "Russia" or "Russian," in any script. For about 4% of neurons, the echo is the whole top of the list. Ablating them across the whole network raises most benchmarks but lowers some reasoning ones.

Attention and the residual stream

Cammarata is more cautious about attention. He writes that he is not sure anyone understands attention well enough to confidently say its illegibility is superposition. He estimates the marginal "explanatory residual" of having global attention head names at about 5%, based on his own guess and an agent's independent estimate across 100 attribution graphs.

On the residual stream, he reports that crushing it four and eight times narrower — down to 46 dimensions in the last block — while keeping all neurons, dropped accuracy from 72% to 60% and 39%. The resulting map looked similar to a normal-width model, just messier. "Nothing looks cleverly interleaved."

Testing explanations that can lose

The essay describes a "blind maximum" test: one agent investigates a neuron and writes a short explanation; a second agent sees only that explanation and writes an example meant to activate it maximally. The result is usually a simple, readable sentence, and it can activate the neuron more strongly than any example found in 10 billion tokens.

In a separate test on 200 random neurons, an agent read only each neuron's explanation and wrote five new examples that should drive it and five near misses that shouldn't. Among the 183 whose explanations describe their positive side, the new examples fired their neurons strongly 76% of the time; the near misses, 11%. The explanation won for 164, tied for 15, and lost for 4.

Agents also wrote the most distant concept they could imagine — the kind of thing a network could pack in without interference. Those texts fired their neurons strongly about 1% of the time, and 179 of the 183 neurons never fired on them at all.

The bitter lesson for interpretability

Cammarata frames the argument as four steps: networks use one learning principle, not two; with modern models and tools that science goes better than expected; understanding compounds; and, speculatively, this might be most of what is needed.

He invokes Rich Sutton's bitter lesson. "Maybe there's a bitter lesson for interpretability too: scaling ordinary investigation beats engineering a clever new decomposition."

He also distinguishes enumerative interpretability, which seeks a big list of features to audit, from exploratory interpretability, which starts with questions and zooms fluidly across attribution graphs and mechanisms. "This approach is far more resilient to polysemanticity."

Criticism and counterpoints from the Hacker News discussion

Several commenters pushed back on the essay's scope. One argued the claims are an overreach, citing the Lottery Ticket Hypothesis and the Multimodal Gap work (arXiv:2203.02053), which suggests networks tend toward condensed representations that are not semantic.

"The claims seem to be an overreach of what is actually proven... I don't think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation."

Another commenter questioned whether the remaining unexplained neurons might be the most important ones:

"If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks."

Neel Nanda's response, linked in the thread, was summarized as: "This is interesting work, but I don't think it's a significant update to the big picture of interpretability." Nanda said he is on board with exploratory interpretability with neurons for practical problems, but does not agree with grander inferences about substantial progress toward ambitious reverse engineering.

A former neuroscientist noted that the primary visual cortex has long been known to contain spatially tuned shape and edge neurons, and asked whether the authors had looked up how the human visual system works.

One commenter argued that the original Zoom In error was treated too casually:

"This type of error may have mislead researchers for years; it was actually harmful. In my field, a correction, or perhaps retraction (depending on the centrality of the claim), would be standard for something of this magnitude."

Others raised the philosophical question of whether a mind can understand a mind much greater than itself, and whether legibility is the same as meaning:

"A neuron is a dirty, contingent, blind-evolved part, like a screw in a particular machine... The essay shows neurons are legible, but legibility isn't the same as meaning."

One commenter offered a compressed summary of the essay's core claim: historically, people thought neurons handle bundles of unrelated concepts, and this blog says that was a methodological error — the dog/car neuron is just a V-shape neuron.

Several commenters praised the essay's presentation, though some noted the interactive 3D visualizations made the page heavy. One reader produced a condensed version with a glossary, and another generated an audio version.

What the essay claims for safety

Cammarata's closing argument is that deception has no reason to hide from a model's own circuitry, because it needs to be accessible to the model's readers. "My fear is that deception will be right there, in plain sight, and we won't know how to read it."

He argues the mistakes are asymmetric. Assuming a network is clean and being wrong makes a messy neuron a lead to investigate. Assuming it is packed and being wrong makes a messy neuron the end of inquiry. "I made that second assumption for years, and it explained away the evidence that would have overturned it."

His guess is that frontier networks are where the vision models were eight years ago: legible, but waiting on many small fixes. "They may be able to do in months in frontier language models what took humans most of a decade for vision models."

A technical companion paper is promised with more detail on language-model neuron experiments, including the different regimes of modern non-linearities and circuit motifs at different depths.