OpenAI Activation Atlases

OpenAI, in collaboration with Google researchers, has developed Activation Atlases, a technique for visualizing the interactions between neurons in a neural network. This method allows researchers to move beyond studying individual neurons to understand the joint representation space of multiple neurons, providing a way to peer into the "black box" of AI decision-making.

Mapping Neural Interactions

Activation atlases expand upon existing feature visualization techniques. While earlier work primarily focused on what individual neurons represent, activation atlases collect and visualize hundreds of thousands of examples of neurons interacting. This shift enables the visualization of the shared space that these neurons jointly represent, offering a more comprehensive view of the network's internal representations.

Improving AI Auditability and Safety

Understanding the internal mechanisms of neural networks is critical for auditing models and ensuring safety in high-stakes deployments. Unlike traditional software, where developers can review code paths or perform formal verification, neural networks lack a similar level of transparency. Activation atlases address this by allowing humans to identify:

• Spurious Correlations: Identifying instances where a network relies on non-causal correlations to classify images.
• Feature Overloading: Discovering where the reuse of a single feature across two different classes leads to bugs.
• Model Vulnerabilities: Enabling humans to design attacks to fool the model based on an understanding of its internal logic.

Case Study: Frying Pans vs. Woks

To demonstrate the utility of activation atlases, researchers created a specific atlas to analyze how a network distinguishes between frying pans and woks. The analysis revealed that while the network correctly identified physical differences (such as frying pans being squarish and woks being rounder and deeper), it also relied on a spurious correlation: the presence of noodles.

Researchers found that adding noodles to the corner of an image could fool the model into classifying the object as a wok 45% of the time. Other human-designed attacks targeting feature detector overloading were even more effective, succeeding up to 93% of the time. This example highlights how models can pick up on correlations that are not causal, a pattern that mirrors concerns regarding fairness and bias in AI systems.

Implications for Interpretability

The success of activation atlases suggests that neural network activations can be meaningful to humans. This finding increases optimism that strong interpretability in vision models is achievable, potentially moving AI systems away from being opaque "black boxes" toward transparent, auditable systems.