Extracting Concepts from GPT-4

OpenAI has introduced new scalable methods to decompose the internal representations of GPT-4 into 16 million potentially human-interpretable patterns, known as "features." This research represents a significant step toward understanding the inner workings of large language models (LLMs) and improving AI safety through interpretability.

The Challenge of Neural Network Interpretability

Neural networks are not designed directly; instead, the algorithms that train them are designed. This creates a "black box" problem where the resulting networks cannot be easily decomposed into identifiable parts, making it reasoning about AI safety fundamentally different from reasoning about traditional engineering safety (such as car safety).

To interpret these models, researchers must find the basic building blocks of neural computations. The difficulty lies in two main areas:

1. Dense Activations: Neural activations in language models typically fire on every input and represent many concepts simultaneously.
2. Sparsity of Real-World Concepts: In any given context, only a small fraction of all possible concepts are relevant.

Sparse autoencoders are used to address this by identifying a small set of features that are important for a specific output, aligning the dense neural activity with sparse activation patterns that humans can more easily understand.

Large-Scale Autoencoder Training and Results

OpenAI developed new methodologies that allow sparse autoencoders to scale to tens of millions of features on frontier AI models. This approach demonstrates smooth and predictable scaling with better returns than previous techniques.

Using this recipe, OpenAI trained autoencoders on both GPT-2 small and GPT-4 activations. The GPT-4 autoencoder identified 16 million features. To verify interpretability, OpenAI visualized these features by analyzing documents where they activate. Examples of identified interpretable features include:

• Human Imperfection: Phrases relating to humans or things being flawed.
• Price Increases: Patterns related to economic shifts.
• X and Y: Mathematical or coordinate-based patterns.
• Training Logs: Technical logs from software training.
• Rhetorical Questions: Linguistic structures used for effect.
• Algebraic Rings: Specific mathematical concepts.
• Who/What the Dopamine: Biological or chemical concepts.

Current Limitations

While this research is a milestone, OpenAI notes several critical limitations:

• Interpretability Gaps: Many discovered features remain difficult to interpret, showing no clear pattern or exhibiting spurious activations unrelated to the concept they encode.
• Incomplete Coverage: The sparse autoencoder does not capture all model behavior. Passing GPT-4 activations through the autoencoder results in performance equivalent to a model trained with roughly 10x less compute.
• Computational Scale: Fully mapping the concepts in frontier LLMs may require scaling to billions or trillions of features, which remains a challenge.
• Functional Understanding: Identifying features at one point in the model is only a single step. Further research is needed to understand how the model computes these features and how they are used downstream.

Future Directions and Open Source Contributions

OpenAI aims to use these features for the practical monitoring and steering of language model behaviors in frontier models. The ultimate goal is to use interpretability to provide strong assurances about model behavior, increasing trust in AI robustness and safety.

To support the research community, OpenAI has released the following resources:

• Research Paper: A detailed account of experiments and methods.
• Code: The code for using the autoencoders.
• Full Suite of Autoencoders: A complete set of autoencoders for GPT-2 small.
• Feature Visualizer: A tool to explore GPT-2 and GPT-4 features.