Anthropic Open-Sources Circuit-Tracing Tools for LLM Interpretability

Anthropic has open-sourced a new method and accompanying library for tracing the "thoughts" of large language models (LLMs) through the generation of attribution graphs. This release aims to bridge the gap between AI capability progress and the understanding of AI inner workings by providing the community with tools to visualize and analyze model circuits.

Attribution Graphs for Model Interpretability

Anthropic's circuit-tracing approach utilizes attribution graphs to partially reveal the internal steps a model takes to arrive at a specific output. By mapping these internal paths, researchers can move beyond treating models as black boxes and instead identify the specific circuits responsible for certain behaviors.

Tooling and Ecosystem

The release consists of two primary components designed for different levels of technical engagement:

• Circuit-Tracer Library: An open-source code repository available on GitHub that supports the generation of attribution graphs for popular open-weights models. This library is intended for sophisticated research and programmatic usage.
• Neuronpedia Frontend: An interactive frontend hosted by Neuronpedia that allows users to generate and explore attribution graphs for their own chosen prompts without needing to manage the underlying code.

Research Capabilities and Applications

The open-source tools enable researchers to perform three primary functions:

1. Circuit Tracing: Generating attribution graphs on supported models to identify internal pathways.
2. Visualization and Collaboration: Using the interactive frontend to visualize, annotate, and share the resulting graphs.
3. Hypothesis Testing: Modifying feature values to observe how those changes directly impact model outputs, allowing for the empirical testing of interpretability hypotheses.

Anthropic has already applied these tools to study multi-step reasoning and multilingual representations in models such as Gemma-2-2b and Llama-3.2-1b. The Gemma graphs specifically leverage transcoders trained as part of the Google GemmaScope project.

Development and Collaboration

This project was led by participants in the Anthropic Fellows program (Michael Hanna and Mateusz Piotrowski), with mentorship from Emmanuel Ameisen and Jack Lindsey. The integration with Neuronpedia was implemented by Decode Research, led by Johnny Lin and Curt Tigges.