hijohnnylin/neuronpedia

open source interpretability platform 🧠

What it solves

Neuronpedia is an open-source interpretability platform designed to help researchers and developers understand the internal workings of neural networks. It provides a centralized hub for exploring, analyzing, and interpreting the features of Sparse Autoencoders (SAEs) and other model activations.

How it works

The platform operates as a suite of standalone services that can be run locally or in the cloud. It integrates several specialized tools for mechanistic interpretability:

  • Inference Server: Handles model execution, supporting backends like vLLM and PyTorch to perform activation testing and steering.
  • Autointerp Server: Uses EleutherAI's Delphi to automatically generate explanations and scores for neural network features.
  • Graph Server: Built on circuit-tracer, it generates attribution graphs to visualize how information flows through a model's circuits.
  • Webapp: A frontend and API that provides dashboards, search capabilities (using pgvector for semantic search of explanations), and data export/upload tools.

Who it’s for

It is primarily for AI safety researchers, mechanistic interpretability practitioners, and developers who want to analyze the internal representations of LLMs.

Highlights

  • Comprehensive Toolset: Includes support for steering, activations, circuits/graphs, and Jacobian lenses.
  • Semantic Search: Allows users to search for features based on the meaning of their natural language explanations.
  • Automated Interpretation: Integrates auto-interpretation and scoring of features to speed up the analysis process.
  • Extensible Architecture: Supports adding custom SAEs and models via Hugging Face and SAELens integration.

Related

  • Dispatch
    Anthropic Open-Sources Circuit-Tracing Tools for LLM Interpretability

    Anthropic has released an open-source library and interactive frontend to generate attribution graphs, allowing researchers to trace the internal decision-making processes of large language models.

  • Project
    omnimind-ai/OmniInfer

    A high-performance, cross-platform inference engine for running LLMs and VLMs locally on desktop and mobile devices with multi-backend support.

  • Dispatch
    Anthropic Interpretability Research Overview

    Anthropic's Interpretability team focuses on understanding the internal workings of large language models to establish a foundation for AI safety and prevent harmful behaviors.

  • Project
    TransformerLensOrg/TransformerLens

    A library for mechanistic interpretability that allows researchers to reverse engineer generative language models by accessing and manipulating their internal activations.

  • Project
    open-webui/open-webui

    A feature-rich, self-hosted AI platform that provides a user-friendly interface for local and API-based LLMs, featuring built-in RAG and extensive plugin support.