anthropics/jacobian-lens

What it solves

It provides a way to understand what a language model is "thinking" internally by translating hidden activations from any layer into actual vocabulary tokens. This allows researchers to see what concepts or words the model is processing at specific positions and layers, even if those words never appear in the input text.

How it works

The tool uses a "Jacobian lens," which is a linear transport mechanism. It calculates the average input-output Jacobian over a text corpus to map a residual-stream vector from an intermediate layer into the final-layer basis. Once transported, the vector is decoded using the model's own unembedding layer to produce a ranked list of likely tokens.

Who it’s for

It is designed for AI researchers and developers interested in mechanistic interpretability and understanding the internal representations of open-weights decoder transformers.

Highlights

• Internal Readout: Decodes internal activations into human-readable tokens.
• Layer-by-Layer Analysis: Supports applying the lens at any layer and position to track how representations evolve.
• Interactive Visualization: Includes a tool to render a layer-by-position view with rank-tracking charts and heatmaps.
• Custom Fitting: Allows users to fit a new lens to their own models using a provided set of prompts.