Why Compression and Large Language Models Solve the Same Prediction Problem

Compression and LLMs are the Same Prediction Problem

Takeaway: Modern lossless compressors and large language models (LLMs) both build probabilistic models of data, use those models to predict the next symbol, and encode the data with an entropy coder; the better the prediction, the fewer bits are needed.

How Classical Compression Works

Transforms (e.g., run‑length encoding) rearrange data to expose redundancy but may not shrink the file directly.
Models assign a probability to each possible symbol based on observed frequencies or context. For the string AAAAAA BBB C, a simple model yields probabilities A:0.71, B:0.21, C:0.07.
Entropy coders (arithmetic or Huffman) convert those probabilities into a bitstream whose length approaches the Shannon entropy -∑p·log₂p.

"Entropy is the floor – the smallest number of bits per symbol achievable without loss." – Ngrok blog

Why Probabilities Matter

Higher‑probability symbols require fewer bits (bits = -log₂(p)).
Skewed distributions compress better: a string dominated by A (p≈0.83) averages 0.82 bits/symbol, whereas a balanced string averages 1.38 bits/symbol.
Adding context (order‑1, order‑2 models) sharpens probabilities dramatically. An order‑1 model reduced the compressed size of the phrase "TO BE OR NOT TO BE" from ~47 bits to ~21 bits.

Language Modeling as Compression

An LLM generates a probability distribution over the next token given the previous context—exactly the model step in a compressor.
When the true next token aligns with the highest‑probability prediction, the encoder spends only -log₂(p) bits; a mis‑prediction inflates the bit cost.
Arithmetic coding can be applied to the token stream, turning a well‑trained LLM into a near‑optimal compressor. In a test on a Dickens quote, a GPT‑2 model achieved 176 bits (10 % of the original), far better than a naïve order‑1 model (434 bits, 24 %).

"Training an LLM is equivalent to optimizing a massive, parametrized compression algorithm (cross‑entropy loss is the same formula as Shannon entropy)." – Ngrok blog

Practical Limits

Model size vs. payload: Deploying a multi‑gigabyte LLM to compress HTTP responses would outweigh the kilobytes saved.
Compute cost: Encoding/decoding with a transformer is orders of magnitude slower and more energy‑intensive than gzip or Brotli.
Therefore, LLMs are useful for semantic compression (e.g., summarizing prompts) but not for routine byte‑level compression.

Community Perspectives

farfatched: "Information theory, inference, and learning are two sides of the same coin; brains are ultimate compressors."
ssivark: "Compression equals prediction only when the training distribution exactly matches the test distribution; otherwise generalization diverges."
variadix: "Non‑LZ compressors implicitly model a probability distribution; the length of each emitted symbol corresponds to its probability."
throwaway_7274: "Viewing training as optimization over a family of compressors makes the emergence of novel ideas plausible."
zephen: "Compression requires prediction, but that does not mean compression is prediction; the directionality matters."
j-pb: "A third facet—indexing—completes the trinity of compression, prediction, and lookup structures."
rrherr: "Schmidhuber’s 2008 paper already linked compression progress to curiosity and creativity, predating recent hype."
sethev: "The Hutter Prize explicitly treats compression as a proxy for intelligence, reinforcing the compression‑prediction link."
sigbottle: "Solomonoff induction shows that perfect compression is vacuous without resource constraints; real compressors must be small enough to be useful."
Lerc: "Prediction enables encoding only the errors, but compressors can also exploit global patterns not visible sequentially, so the equivalence is not strict."
zhxiaoliang: "Compression exploits predictability; intelligence creates useful predictions—an important conceptual distinction."
jdthedisciple: "Predictability is the inverse of information; low information yields high compression, which is basic information theory."
e12e: "Gzip can be used as a language model, demonstrating that classic compressors already perform next‑token prediction."

Bottom Line

Mathematical identity: Both lossless compressors and LLMs minimize the same objective—cross‑entropy (negative log‑likelihood), which equals the expected bits per symbol.
Practical divergence: Compressors are engineered for speed, tiny models, and deterministic decoding; LLMs prioritize expressive power and can handle long‑range dependencies but at huge resource cost.
Future direction: Improving compression hinges on better predictive models. As LLMs continue to advance, they will increasingly serve as semantic compressors, while traditional byte‑level compressors will remain the workhorse for low‑overhead data transmission.

Quick Reference Table

Component	Compression	LLM	Role
Model	Probability table (often context‑aware)	Transformer that outputs token probabilities	Predict next symbol
Entropy coder	Arithmetic/Huffman → bitstream	Same coder can be applied to token probabilities	Convert probabilities to bits
Goal	Minimize bits / symbol (approach entropy)	Minimize cross‑entropy loss (same metric)	Better prediction → better compression

In summary, the core of any lossless compressor is a predictor; modern LLMs are simply far more powerful predictors, which is why they achieve superior compression when paired with an entropy coder, even though practical constraints keep them separate in most engineering pipelines.