inseq-team/inseq
Interpretability for sequence generation models 🐛 🔍
What it solves
Inseq is a toolkit designed to make post-hoc interpretability analysis of sequence generation models (like LLMs and translation models) more accessible. It allows users to understand why a model generated a specific token by attributing the output to specific parts of the input text.
How it works
Built on PyTorch and integrating with Hugging Face Transformers, Inseq provides a unified interface to apply various feature attribution methods. It supports encoder-decoder models (ForConditionalGeneration) and decoder-only models (ForCausalLM). The toolkit includes a variety of attribution techniques—gradient-based, internals-based (attention), and perturbation-based—and provides tools to aggregate, filter, and visualize these attribution maps in notebooks, browsers, or the command line.
Who it’s for
It is primarily aimed at AI researchers and developers who need to analyze the behavior of generative models, perform contrastive evaluations, or detect context reliance in their AI applications.
Highlights
- Broad Model Support: Works with most Hugging Face Transformers generation models.
- Diverse Attribution Methods: Includes saliency, Integrated Gradients, DeepLIFT, LIME, and occlusion.
- Advanced Analysis: Supports contrastive feature attribution and context reliance detection.
- Flexible Visualization: Built-in support for heatmaps in Jupyter Notebooks and HTML exports.
- CLI Tools: Command-line interface for attributing single examples or entire Hugging Face datasets.