Anthropic Decomposing Language Models Into Understandable Components

Anthropic has introduced a method to decompose language models into interpretable components called features, enabling a deeper understanding of how neural networks process information. This approach addresses the "black box" nature of AI by identifying linear combinations of neuron activations that correspond to specific, consistent concepts rather than relying on individual neurons, which often lack a consistent relationship to model behavior.

Moving Beyond Individual Neurons to Features

Individual neurons in neural networks are often polysemantic, meaning a single neuron may activate for multiple unrelated contexts. For example, a single neuron in a small language model might activate for academic citations, English dialogue, HTTP requests, and Korean text simultaneously. Because individual neurons do not have a consistent relationship to network behavior, they are difficult to use for diagnosing failure modes or certifying model safety.

To solve this, Anthropic researchers identified "features" as the primary unit of analysis. Features are patterns or linear combinations of neuron activations. By using dictionary learning, the researchers were able to decompose a layer of 512 neurons into more than 4,000 features. These features represent distinct concepts such as:

• DNA sequences
• Legal language
• HTTP requests
• Hebrew text
• Nutrition statements

Most of these model properties are invisible when analyzing the activations of individual neurons in isolation.

Validation of Feature Interpretability

Anthropic validated the interpretability of these features using two primary methods:

Human Evaluation

Blinded human evaluators scored the interpretability of both neurons and features. The results showed that features received significantly higher interpretability scores than individual neurons.

Autointerpretability

Researchers used a large language model (LLM) to generate short descriptions of the features in the small model. These descriptions were then scored based on the ability of another model to predict a feature's activations based solely on that description. Features again scored higher than neurons, confirming that feature activations and their downstream effects on model behavior have a consistent interpretation.

Model Steering and Generalization

Features provide a mechanism for targeted model steering. Artificially activating a specific feature causes the model's behavior to change in predictable ways, allowing researchers to manipulate model output based on internal representations.

Furthermore, the research indicates that these learned features are largely universal across different models. This suggests that the lessons learned from studying features in one model may generalize to other models. The researchers also found that the number of features learned can act as a "knob" to vary the resolution of the model's internal view: a small set of features provides a coarse, easier-to-understand view, while a larger set reveals more subtle model properties.

Implications for AI Safety and Scaling

This work is part of Anthropic's investment in Mechanistic Interpretability, a long-term research bet on AI safety. By overcoming the roadblock of uninterpretable neurons, this method provides a path toward monitoring and steering model behavior from the inside, which is essential for the reliability and reliability required for enterprise and societal adoption.

The current success has been demonstrated on a small model. The next primary challenge is scaling this approach to frontier models, which are significantly larger and more complex. Anthropic states that the primary obstacle to interpreting large language models is now an engineering challenge rather than a scientific one.