LLM Attention Visualization Tool Shows How Tokens Influence Generation

Quick Take

The LLM Attention Visualizer lets you hover over any generated token and instantly see a weighted map of past tokens that contributed to it, exposing the model’s copy‑and‑paste behavior and multi‑phrase synthesis.

What the Visualizer Shows

  • Attention aggregation: For each generated token the tool computes the attention weight (scaled by the value‑vector magnitude), sums it across all heads and layers, and maps the result to an opacity value for each previous token.
  • Opacity meaning: An opacity of 1 indicates the strongest influence; lower opacities are interpolated proportionally.
  • Simplification disclaimer: The visualizer reduces a high‑dimensional attention tensor to a single scalar per past token, so it does not capture the full nuance of the model’s internal state.

"Affected" might not be fully accurate, as this visualization is highly simplified. It's calculating the attention weight, scaled by the magnitude of the value vector, aggregated across all attention heads, and summed across all layers. – author’s note

Why It Matters

  • Copy‑paste intuition: Hovering over verbatim strings (e.g., addresses or dates) highlights the original source tokens, explaining why LLMs can reproduce text with near‑zero error despite their probabilistic nature.
  • Cross‑phrase synthesis: In the "Office Move Summary" example, the token remain draws attention from both work (in "Existing employee access cards will work") and stay the same (in "company phone numbers will stay the same"), demonstrating how the model blends information from separate clauses.
  • Model size relevance: Even a 600 M‑parameter model (Qwen‑3‑0.6B) can faithfully reproduce a JavaScript function, showing that precise copying does not require billions of parameters.

Implementation Details

  • Frontend: A React app powered by Transformers.js runs inference directly in the browser.
  • Custom generation loop: The standard generation API does not expose intermediate tensors, so the author wrote a bespoke loop to capture attention scores at each step.
  • Model instrumentation: The original ONNX graph lacks the needed outputs. The author patched the ONNX file to expose internal tensors, then uploaded the instrumented model to a private Hugging Face repo (ishamf/Qwen3-0.6B-ONNX-Instrumented).
  • Pre‑generated prompts: Because the model (hundreds of MB) takes time to download, the app ships with a set of pre‑computed generations that load instantly.
  • Source code: Available at the GitHub repository https://github.com/ishamf/llm-visualizer.

Community Reactions

  • Clarity for learners – "This is the clearest example I've seen on how attention works" (fuddle).
  • Pedagogical value – "Having a visualization like this helps a lot. I'm teaching this on Friday" (MCP123).
  • Technical curiosity – "Are you worried later‑layer attention gets drowned out by earlier layers because there are more of them contributing to the sum?" (wopak).
  • Critique of simplification – "I highly question this simplistic idea of high vector magnitude = high influence" (sva_).
  • Performance question – "Does that mean you have N² computation with the context size since every token needs to track how it relates to every other token?" (ex‑aws‑dude).
  • UX suggestions – A user noted the animation runs too fast and suggested pause/step controls for detailed inspection (mncharity).

Limitations and Open Questions

  • Scalar reduction: Collapsing multi‑head, multi‑layer attention to a single opacity discards directionality and sign information.
  • Layer weighting: The current sum treats all layers equally; alternative weighting schemes could surface deeper‑layer patterns that are otherwise muted.
  • Magnitude bias: Scaling by value‑vector magnitude assumes larger vectors imply greater influence, a point contested by some commenters.
  • Scalability: Visualizing attention for very long contexts (e.g., >2 k tokens) would still require O(N²) computation and may become unwieldy in a browser.

How to Use It

  1. Open the visualizer at https://ishamf.dev/p/llm-attention-visualizer/.
  2. Hover over any token in the generated text.
  3. Observe the opacity‑graded list of past tokens; the brightest tokens contributed the most.
  4. Switch between pre‑generated prompts to explore different behaviors (copying, synthesis, function modification, etc.).

Takeaway

The LLM Attention Visualizer provides an immediate, interactive window into the attention dynamics that enable large language models to copy exact strings and blend information from disparate phrases, making abstract transformer internals concrete for educators, researchers, and curious developers.

Sources

Related

  • Dispatch
    Demystifying the Transformer: A Deep Dive into the Math of Modern LLMs

    An exploration of the Transformer Math Explorer tool, breaking down the fundamental mathematical operations of LLMs from GPT-2 to Qwen, including attention mechanisms and KV caching.

  • Dispatch
    DiffusionGemma: The First Diffusion LLM Natively Supported in vLLM

    vLLM has integrated DiffusionGemma, a 26B-parameter discrete diffusion language model, enabling high-throughput, low-latency generation by iteratively denoising token blocks instead of sequential autoregressive decoding.

  • Project
    zai-org/GLM-5

    GLM‑5 is an open‑weight LLM family (up to 744 B parameters, 1 M‑token context) focused on coding, cyber‑security, and long‑horizon agentic tasks. It introduces sparse‑plus‑linear attention (Flash), the IndexShare efficiency trick, and a reasoning‑effort API. Models are downloadable from Hugging Face/ModelScope and can be served with SGLang, vLLM, Transformers, KTransformers, or Ascend NPU. Fine‑tuning is supported via the slime RL framework and ms‑swift. The series claims state‑of‑the‑art open‑source performance on several benchmarks.

  • Project
    poloclub/transformer-explainer

    An interactive visualization tool that runs a live GPT-2 model in the browser to help users learn how Transformer-based models predict text.

  • Dispatch
    Optimizing LLMs in Production: Precision, Attention, and Architecture

    Hugging Face outlines key techniques for efficient LLM deployment, focusing on lower precision quantization, Flash Attention for memory efficiency, and architectural optimizations like RoPE, ALiBi, MQA, and GQA.