Why the Next-Token Predictor Mental Model is Incomplete for Modern LLMs

The Limitation of the Next-Token Predictor Model

While Large Language Models (LLMs) operate as next-token predictors at inference time—emitting tokens one by one—this description is an incomplete mental model for how modern models are trained and what they encode. The "next-token predictor" label accurately describes the mechanism of output (the shape of the inference loop), but it fails to capture the objective of post-trained models.

Pre-training vs. Post-training Objectives

There is a fundamental distinction between how a base model learns and how a post-trained model evolves through reinforcement learning.

Pre-training: Imitation and Prediction

During pre-training, the model functions as a traditional next-token predictor. It analyzes existing sequences in a training dataset and adjusts its parameters to make the actual next token in that sequence more likely to be sampled. The objective is purely predictive: the model learns to imitate the statistical patterns of its training data.

Post-training: Reward Maximization via RLVR

Modern LLMs undergo post-training, specifically Reinforcement Learning with Verifiable Rewards (RLVR). In RLVR, the model does not have a "correct" next token to predict from a dataset. Instead, it explores by generating new sequences and receives a reward based on the outcome.

In this paradigm, the model makes a token more likely not because it appeared in a training set, but because the sequence containing that token led to a high reward. The objective shifts from predicting the next token in data to maximizing a reward function.

The Chess Analogy: Prediction vs. Optimization

To illustrate the difference between imitation and optimization, consider two types of chess systems:

  1. The Next-Move Predictor: A system trained on a database of grandmaster games. It predicts the move a grandmaster would most likely play in a given position. It is limited to imitating existing human play.
  2. The Reward Maximizer: An idealized engine that explores all possible games to determine the probability of winning from any position. It chooses the move that maximizes the chance of victory, regardless of whether a human ever played that move.

Calling the second system a "next-move predictor" is misleading because its goal is not to predict a dataset, but to win the game.

Synthesis of Community Perspectives

Discussion among technical practitioners reveals a deep divide on whether the "next-token predictor" label is fundamentally wrong or simply a matter of abstraction level.

Arguments for the Model's Validity

Some argue that regardless of the training objective (RLVR or pre-training), the mathematical operation remains the same: calculating the conditional probability $P(\text{token}k | \text{tokens}{1...k-1})$.

"The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think."

Others suggest that the "next-token predictor" model is useful for explaining common LLM failure modes, such as "locking in" to a wrong answer early in a response because the model cannot self-edit previously emitted tokens.

Arguments Against the Model's Validity

Critics of the label argue that it is often used as a "reductionist" tool to dismiss the emergent capabilities of LLMs. They contend that focusing on the token-by-token mechanism ignores the complex internal representations and "world models" that must be formed to make those predictions accurate.

"It's like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level of simplification we've just ignored 150 tons of advanced engineering and physics."

Emergence and Agentic Systems

Some suggest that the "next-token predictor" is the base component, but the resulting system is something more. In this view, modern AI is an emergent system where simple rules (token prediction) and complex interactions (RLVR, tool use, recursive invocation) create capabilities that transcend the simple label of a predictor.

Sources

Related

  • Dispatch
    AI Reasoning and the Illusion of Thinking: Are Large Reasoning Models Right for the Wrong Reasons?

    Research into Large Reasoning Models (LRMs) suggests that their 'chains of thought' may be unfaithful representations of internal processes, functioning more as probabilistic anchors than logical steps.

  • Project
    LeapLabTHU/limit-of-RLVR

    A research project and evaluation framework that analyzes whether Reinforcement Learning with Verifiable Rewards (RLVR) expands an LLM's reasoning capacity or merely optimizes its sampling efficiency.

  • Project
    mll-lab-nu/VAGEN

    VAGEN is a reinforcement learning framework for training multi-turn VLM agents that improves performance by reinforcing their internal world model through explicit visual state reasoning.

  • Dispatch
    The Rent-Seeking Loop: Beyond the 'Next-Token Prediction' Debate

    An exploration of the economic and social implications of LLMs, arguing that the real danger isn't the technology itself, but the concentration of the means of production in the hands of a few.

  • Dispatch
    Prompt Injection as Role Confusion

    Research reveals that LLMs identify roles based on writing style rather than structural tags, allowing attackers to bypass security via 'CoT Forgery' and other style-based spoofing.