Stanford CS229 Spring 2026 Lecture 16: Transformer Attention Variants, Mixture of Experts, and In‑Context Learning

Transformer Attention Recap

The lecture begins by reminding the audience of the basic transformer attention mechanism. For a single head, queries Q, keys K, and values V are matrices where each row corresponds to a time step. Attention scores are obtained by QKᵀ, masked to enforce autoregressive behavior, followed by a row‑wise softmax and multiplication by V. The result is the weighted sum of values. When multiple heads are used, each head has its own Q, K, V matrices with different weights but the same structure.

Group Query Attention

To lower the memory footprint of the key‑value (KV) cache during inference, the lecture describes group query attention (GQA). Instead of storing a separate key and value vector for every head, GQA shares a smaller set of keys and values across multiple query heads. Each query head is assigned to a key group via a deterministic mapping (e.g., floor((j‑1)/tall)+1). The attention computation uses the shared key for all queries in the group, while each head still produces its own output. This reduces the number of stored key/value vectors from NH to NG, where NH is the number of heads and NG the number of key groups, thereby decreasing GPU memory usage and allowing larger batch sizes.

Sliding Window Attention

The lecture then presents sliding window attention as a way to cut the quadratic compute cost of full attention. Rather than letting each query attend to all previous keys, each query attends only to the most recent W keys, where W is a fixed window size. This changes the complexity from O(T²) to O(T·W). Although the top‑layer attention is limited to recent tokens, deeper layers can indirectly propagate information from earlier tokens, giving an effective receptive field of roughly L·W tokens for L layers. The trade‑off is reduced long‑range dependency versus lower compute and memory.

Mixture of Experts

Next, the instructor covers mixture‑of‑experts (MoE) layers as a method to increase model parameter count without a proportional rise in compute. An MoE layer contains many expert networks (each typically a feed‑forward network) and a routing module that selects a small subset of experts for each token. For example, with 128 total experts and a top‑2 routing, only 2 experts are active per token, giving an active parameter count far smaller than the total. The routing is usually based on a learned projection followed by a softmax or sigmoid, and the selected experts’ outputs are combined via a weighted sum that is renormalized to sum to one. Shared experts (always active) can be included to capture common knowledge. The motivation is both computational efficiency and the hope that experts specialize to different linguistic phenomena, although specialization emerges implicitly from end‑to‑end training rather than explicit supervision.

In‑Context and Zero‑Shot Learning

The lecture explains how large language models can be adapted to new tasks without gradient updates. In‑context learning involves concatenating demonstration examples (input‑output pairs) with a test input and letting the model generate the answer. Zero‑shot learning goes further by providing only a task description (e.g., "classify emails into billing or technical)) and no examples. The model relies on its pretrained knowledge to infer the correct behavior. These approaches work because the model treats the prompt as context and generates continuations that match the demonstrated pattern or follow the instruction.

Instruction Tuning (Supervised Fine‑Tuning)

Finally, the instructor describes instruction tuning as a way to strengthen zero‑shot and in‑context abilities. A dataset of instruction‑output pairs (X, Y) is collected, where X is a task description and Y is the desired response. The model is then further trained on this dataset using the standard language‑model loss (negative log likelihood of Y given X). This supervised fine‑tuning does not change the architecture; it adjusts the parameters so that the model is more likely to follow instructions and produce appropriate outputs when prompted. The resulting model exhibits improved performance on both zero‑shot and few‑shot tasks.