lyramakesmusic/neuron-explorer

What it is

A tool for exploring the internal "neurons" of a large language model (Qwen3.5-4B). It systematically tests what happens when each neuron is artificially boosted or suppressed during text generation, saves every experiment, and displays the results in an interactive 3D view of the model.

How it works

The project has two halves:

1. The sweep — a batch pipeline that runs on a GPU server. It pushes every neuron in a layer up or down by measured amounts, generates text, and records the outputs.
2. The explorer — a local web app that loads the model on your own machine and lets you browse all the sweep results, run live experiments, and chat with an AI assistant about what you find.

The Explorer (web app)

Runs at localhost:8791. It reads the sweep's output files and loads the model locally (Apple silicon, NVIDIA GPU, or CPU) for live interaction. It has five panels:

• Map — a 3D view of the model's layers; each neuron is a point. Hovering shows what that neuron steers toward and what token it predicts; clicking opens its details. Neurons with similar internal structure are placed next to each other.
• List — neurons ranked by how strongly they cluster around specific words, compared against random baselines. You can search for a token, or run your own text and see which neurons activate.
• Neuron — for a single neuron: steered text generations at different doses, a three-way chat (steer down / neutral / steer up), its connections to other neurons, and a name.
• Embeddings — a UMAP projection of a layer's experiments, colored by neuron, prompt, dose, or kind.
• Claude — a chat that uses the claude command-line tool (if installed and logged in), with the explorer acting as a tool server. Claude can read neuron data, run the live model (steering, activations, next-token probabilities, ablations), navigate the map and lists, and use file and shell tools. It only answers on your own machine.

The Sweep (data pipeline)

Runs on a CUDA GPU pod using a modified version of the vLLM inference engine. Key steps:

• Setup — installs the modified vLLM, dependencies, and the model.
• Neuron stats — computes per-neuron statistics: activation extremes, down-projection columns, residual norms.
• Main sweep — for every neuron, generates text with the neuron pushed up or down. Doses are fractions of the median residual norm leaving the layer: three prompts ("who", "snake", "orange") each at three doses (−0.7, +0.5, +0.8), one "free" condition at +3, plus one unsteered baseline per prompt.
• Depth sweep — a random-subset sweep across several layers.

Supporting scripts produce rankings (distinctive words, cluster mass vs. controls), "steer lenses" (how each neuron changes next-token probabilities), per-layer depth reports, word-shift scores, merged shards, equivalence checks between two inference engines, throughput benchmarks, and the cached data the explorer uses (embeddings, layout, connections — each neuron's 24 strongest links up and down).

Notes

research_log.md holds the project's findings and reasoning; NOTES.md holds source notes.