hijohnnylin/neuronpedia

What it solves

Neuronpedia is an open-source interpretability platform designed to help researchers and developers understand the internal workings of neural networks. It provides a centralized hub for exploring, analyzing, and interpreting the features of Sparse Autoencoders (SAEs) and other model activations.

How it works

The platform operates as a suite of standalone services that can be run locally or in the cloud. It integrates several specialized tools for mechanistic interpretability:

• Inference Server: Handles model execution, supporting backends like vLLM and PyTorch to perform activation testing and steering.
• Autointerp Server: Uses EleutherAI's Delphi to automatically generate explanations and scores for neural network features.
• Graph Server: Built on circuit-tracer, it generates attribution graphs to visualize how information flows through a model's circuits.
• Webapp: A frontend and API that provides dashboards, search capabilities (using pgvector for semantic search of explanations), and data export/upload tools.

Who it’s for

It is primarily for AI safety researchers, mechanistic interpretability practitioners, and developers who want to analyze the internal representations of LLMs.

Highlights

• Comprehensive Toolset: Includes support for steering, activations, circuits/graphs, and Jacobian lenses.
• Semantic Search: Allows users to search for features based on the meaning of their natural language explanations.
• Automated Interpretation: Integrates auto-interpretation and scoring of features to speed up the analysis process.
• Extensible Architecture: Supports adding custom SAEs and models via Hugging Face and SAELens integration.