EleutherAI/sparsify

What it solves

It provides a lean library for training and loading k-sparse autoencoders (SAEs) and transcoders on the activations of HuggingFace language models. It is designed to scale to very large models and datasets by computing activations on-the-fly rather than caching them to disk, eliminating storage overhead.

How it works

The library uses a TopK activation function to directly enforce sparsity in the activations, which the authors believe is a Pareto improvement over L1 penalty methods. Users can train SAEs on the residual stream or custom hookpoints (using Unix pattern matching) across different submodules of a model. It supports distributed training via PyTorch's torchrun, with an option to distribute modules across GPUs to save memory.

Who it’s for

Researchers and developers working on mechanistic interpretability who need to train SAEs on large-scale language models without being limited by storage constraints.

Highlights

• On-the-fly activation computation: Reduces storage overhead for large datasets.
• TopK activation function: Directly enforces sparsity levels.
• Flexible hookpoints: Supports Unix pattern matching for targeting specific model layers or modules.
• Distributed training: Includes memory-efficient module distribution across GPUs.
• Pretrained SAEs: Ability to load pretrained SAEs directly from the HuggingFace Hub.