OpenAI releases automated neuron explanation dataset for GPT‑2 using GPT‑4

TL;DR

OpenAI introduced an automated pipeline that leverages GPT‑4 to write and evaluate natural‑language explanations of individual neurons in GPT‑2, and open‑sourced the full set of explanations, scores, a visualization tool, and code.

Method Overview

The approach applies three steps to each of the 307,200 neurons in GPT‑2: (1) prompt GPT‑4 to produce a short description of the neuron's behavior, (2) have GPT‑4 generate counter‑examples and refine the description, and (3) score the final explanation using GPT‑4’s own judgment. This pipeline runs on the OpenAI API and is designed to scale interpretability work to models with billions of parameters.

Key Findings

  • Explanation quality varies across layers: Scores drop in later layers of larger models, indicating that deeper neurons are harder to capture with short textual descriptions.
  • Iterative refinement improves scores: Asking GPT‑4 for counter‑examples and revising explanations raises the average quality.
  • Model size matters: Larger explainer models produce higher‑scoring explanations, yet even GPT‑4 underperforms human annotators.
  • Activation function impacts interpretability: Training GPT‑2 variants with different activation functions led to modest score improvements.
  • Dataset statistics: Over 1,000 neurons received scores of at least 0.8, meaning GPT‑4 judged the explanation to cover most of the neuron’s top‑activating behavior. Most high‑scoring neurons are unremarkable, but many interesting neurons remain poorly explained.

Released Resources

  • Explanation dataset: Natural‑language explanations and GPT‑4 scores for every GPT‑2 neuron.
  • Neuron viewer: An interactive web tool for browsing explanations and activation patterns.
  • Codebase: Open‑source implementation of the explanation and scoring pipeline, compatible with publicly available models via the OpenAI API (GitHub: openai/automated-interpretability).

Limitations

  • Simplicity of explanations: Short textual descriptions may not capture polysemantic or highly complex neuron behavior.
  • Scope of analysis: The method explains neuron activations relative to input text only, without addressing downstream effects or circuit‑level interactions.
  • Correlation vs. causation: Explanations describe observed correlations and may fail on out‑of‑distribution inputs.
  • Compute cost: Running GPT‑4 for every neuron is computationally intensive.

Future Directions

OpenAI aims to extend the technique to:

  • Generate explanations for full neural circuits, including attention heads.
  • Produce richer, possibly multi‑sentence or structured, descriptions that capture polysemanticity.
  • Automate hypothesis testing and iteration, mirroring the workflow of human interpretability researchers.
  • Apply the pipeline to larger, more capable models to detect alignment and safety issues such as deception or bias before deployment.

Community Call‑to‑Action

The released dataset and tools are intended to spur research on:

  • New algorithms for higher‑quality neuron explanations.
  • Visualization and analysis methods that leverage the explanations.
  • Benchmarks that compare automated explanations against human interpretations.

“We hope the research community will develop new techniques for generating higher‑scoring explanations and better tools for exploring GPT‑2 using explanations.” — OpenAI research team

Authors: Jan Leike, Jeffrey Wu, Steven Bills, William Saunders, Leo Gao, Henk Tillman, Daniel Mossing

Sources