Anthropic Research: Tracing Model Outputs to Training Data using Influence Functions

Anthropic has developed a method to scale influence functions to large language models (LLMs) with up to 52 billion parameters, allowing researchers to trace specific model outputs back to the most influential training examples. This research demonstrates that as models scale, they shift from relying on superficial token overlaps to generalizing via abstract, thematic concepts.

Scaling Influence Functions to Large Models

Influence functions are a statistical technique used to determine which training examples most significantly contribute to a model's output by treating them as a counterfactual: calculating how the model's parameters and outputs would change if a specific training example were added to the dataset.

Historically, this process was computationally prohibitive for large models because it required computing inverse-Hessian-vector products and gradients for all candidate training examples. Anthropic's research, detailed in the paper "Studying Large Language Model Generalization with Influence Functions," introduces efficient algorithms that enable this analysis on models ranging from 810 million to 52 billion parameters.

The Relationship Between Model Scale and Generalization

Research findings indicate that the patterns of generalization become more abstract as the model size increases. This transition is evidenced by how different sized models identify influential training sequences:

• Small Models (e.g., 810M parameters): Influential sequences often share overlapping tokens or specific words (e.g., "continue existing" or the word "clip") but are otherwise semantically unrelated to the output.
• Large Models (e.g., 52B parameters): Influential sequences are more conceptually related to the output. For example, when a model expressed a desire not to be shut down, the 52B model's influential sequences involved themes of survival instinct and humanlike emotions in AI, rather than just matching keywords.

This trend is also visible in chain-of-thought reasoning for math word problems, where larger models rely on training examples that explain similar reasoning processes rather than those that share superficial vocabulary.

Cross-Lingual Influence and Generalization

Model scale also impacts how influence transfers across languages. When translating a query (such as the anti-shutdown example) into Korean and Turkish, the researchers found that the influence of English-language training sequences on the translated queries became considerably stronger as the model size increased. This suggests that larger models develop a more language-agnostic, conceptual understanding of the information.

Memorization vs. Generalization

Analysis of influence patterns suggests that LLM outputs are not the result of pure token-level memorization. The researchers found that:

• Distribution: Influence typically follows a power law distribution, meaning a small fraction of training data provides most of the influence.
• Diffusion: Despite the power law, influence remains diffuse. The influence of any single training sequence is smaller than the information content of a typical sentence, indicating the model is not simply reciting individual training examples.

Localizing Influence within the Network

Influence functions can be used to map where influence is distributed across the neural network's architecture. While influence is approximately evenly distributed among layers on average, specific queries often show localization:

• Bottom and Top Layers: These layers tend to capture detailed wording and specific token information.
• Middle Layers: These layers generally handle generalization at a more abstract, thematic level.

Implications for AI Alignment and Interpretability

This top-down approach to interpretability complements bottom-up mechanistic interpretability (which studies individual units and circuits). By starting with observable behaviors and digging down to the responsible neurons and circuits, researchers can study high-level cognitive phenomena like reasoning and role-playing that only emerge at scale. Anthropic intends to extend this work to fine-tuning, which is critical for understanding how supervised and reinforcement learning objectives affect model alignment and behavior.