Studying Large Language Model Generalization with Influence Functions

Anthropic has developed a method to scale influence functions to large language models (LLMs) by using the Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) approximation. This allows researchers to identify which specific training examples most contribute to a given model behavior, providing a counterfactual understanding of how model parameters and outputs would change if a specific sequence were added to the training set.

Scaling Influence Functions with EK-FAC

Computing influence functions for LLMs is traditionally difficult because it requires calculating an inverse-Hessian-vector product (IHVP). The EK-FAC approximation allows this computation to be orders of magnitude faster while maintaining accuracy similar to traditional influence function estimators. This method enables the analysis of models with up to 52 billion parameters.

To further reduce the computational cost of calculating gradients for candidate training sequences, Anthropic employs two specific algorithmic techniques:

• TF-IDF filtering: Used to narrow down the candidate training sequences to be analyzed.
• Query batching: Used to optimize the processing of multiple queries simultaneously.

Investigating LLM Generalization Patterns

Using these scaled influence functions, Anthropic investigated several key areas of LLM generalization, including:

• Sparsity of influence patterns: Analyzing how a few training examples may drive specific outputs.
• Abstraction with scale: Observing how the model's ability to abstract information increases as the model size grows. -- Math and programming abilities: Determining the source of the model's ability to perform mathematical reasoning and coding tasks.
• Cross-lingual generalization: Understanding how training on one language affects performance in others.
• Role-playing behavior: Identifying the training data that triggers specific persona adoption.

Limitations in Generalization

Despite the model's showing sophisticated forms of generalization, the research identifies a critical limitation: influence functions decay to near-zero when the order of key phrases in a prompt is flipped. This suggests that the model's understanding of the generalization pattern is highly sensitive to the sequence of information provided in the training data.

Implications for Model Safety and Risk Mitigation

Influence functions provide a powerful tool for the research community to understand the model's internal logic and gain better visibility into the LLM's training process. By identifying the training examples that most influence a specific behavior, developers can better understand and mitigate the associated risks associated with the model's outputs.