w3cj/how-llms-work

What it solves

Most people use LLMs as a black box. This project opens that box: it walks you through every major stage of how a language model actually works, from a silly pattern-matching chatbot to a full transformer trained from scratch. Instead of just reading theory, you get working code you can run locally and watch produce results in real time.

How it works

The app is a set of interactive demos, each following the same three-part architecture: a Hono server route that processes input and streams events, a client hook that manages state and connects to the stream via SSE, and a result component that visualizes what’s happening. The five sections are:

1. A simple chat bot that uses if-statements (ELIZA-style) to show that intelligence isn’t required for the streaming effect.
2. An XOR neural network trained live via backpropagation, proving why multiple layers are needed.
3. A Byte Pair Encoding (BPE) tokenizer implemented from scratch that animates merge steps on your own text.
4. A Word2Vec skip-gram embedding trainer that learns word vectors and shows words clustering by meaning.
5. A decoder-only transformer (GPT-like) trained from scratch — no ML libraries — with multi-head causal self-attention, layer normalization, feed-forward layers, backpropagation, and Adam optimization, using multi-threaded parallelism.

Everything runs in the browser via Vite, and the code is split into server routes, hooks, and components for easy inspection.

Who it’s for

Developers, students, or anyone curious about how LLMs really work under the hood. It’s especially useful if you want to see the math and implementation details without relying on high-level frameworks. Basic familiarity with JavaScript and neural network concepts helps, but the demos make the ideas concrete.

Highlights

• All five LLM stages covered with runnable, local code
• Transformer built entirely from scratch (no ML libraries)
• Live visualizations: animated BPE merges, embedding clustering, training loss
• Same SSE streaming architecture as real ChatGPT plumbing
• References to the original papers (ELIZA, backpropagation, BPE, Word2Vec, Attention Is All You Need, GPT-1, etc.)
• Requires only Node.js 20+ and pnpm