Engram CEO Dan Biderman Explains the AI Memory Problem and Why Long Context Isn't Enough

Engram CEO Dan Biderman Explains the AI Memory Problem and Why Long Context Isn't Enough

TL;DR

Engram’s Dan Biderman says that longer context windows alone cannot give AI systems reliable, long‑horizon memory; instead, we need to compress knowledge into knowledge cartridges and continuously update model weights through efficient continual learning. This approach promises token‑efficient, cost‑effective AI that can handle trillion‑token corpora and personalize to individual users.

1. Why Long Context Is Not Sufficient

  • Context rot: Adding more tokens makes models increasingly confused; even a 10‑million‑token window does not guarantee holistic reasoning.
  • Compaction limits: Current compaction (evicting less‑important tokens) is lossy and can cause forgetting, especially in deep sessions.
  • Memory inefficiency: Loading a few‑kilobyte article into a LLaMA‑70B model consumes ~80 GB of GPU HBM, far exceeding the model’s 140 GB parameter size. The KV‑cache becomes a systems bottleneck.
  • Cost explosion: Querying frontier models over trillion‑token corpora would cost thousands of dollars per query, making RAG impractical at enterprise scale.

2. Knowledge Cartridges: Compressing Corpora into Model State

  • Concept**: Train a model on a large corpus in advance so that the resulting brain state (the cartridge) encodes the knowledge in a highly compressed form—potentially a thousand‑times smaller than raw text.
  • Usage: Load a cartridge into the model at inference time; the model can then reason with far fewer tokens, reducing confusion and cost.
  • Granularity: Cartridges can be task‑specific (e.g., a skill) or corpus‑specific (e.g., a company’s internal documents).
  • Analogy: Like a chef who reads cookbooks and internalizes intuition; the cartridge captures the “intuition” that goes beyond raw recipes.

3. Continual Learning & Test‑Time Training

  • Goal: Update the model’s weights with new data without destroying existing knowledge, achieving token efficiency and enabling longer‑horizon tasks.
  • Method: Perform gradient‑based updates during inference (test‑time training) so the model can absorb fresh information on the fly.
  • Benefit: Allows a single model to handle both static knowledge (stored in cartridges) and dynamic, user‑specific updates.

4. Token Efficiency as a Proxy for Intelligence

  • Premise: Smarter AI does more with fewer compute cycles; token efficiency directly correlates with the ability to solve harder problems.
  • Routing: Engram plans to route queries to the smallest capable model (e.g., cheap open‑source LLM) when possible, reserving larger models for high‑pay‑grade tasks.
  • Future vision: Personal AI agents that continuously improve like a Tamagotchi—users nurture them, and the agents adapt without constant human supervision.

5. Enterprise Use Cases

  • Holistic queries: Tasks that require aggregating information across thousands of files (e.g., “Which M&A deals are incomplete this year?”) cannot be answered by simple RAG; they need a model that has learned the relationships.
  • Cost‑effective scaling: By compressing corporate knowledge into cartridges and using token‑efficient inference, Engram can answer such queries for a fraction of the cost of naïve RAG.

6. Open Research Questions

  • What belongs in weights vs. text? Determining which facts should be internalized (weights) and which should remain external (retrievable text) is an unsolved problem, akin to human memory studies.
  • Automatic saliency detection: Engram is training models to decide autonomously what to store in cartridges, avoiding manual heuristics that become a "whack‑a‑mole" problem.
  • Scalable infrastructure: Deploying millions of personalized cartridges across devices will require new APIs, storage formats, and inference pipelines.

7. Team & Culture

  • Engram’s research team blends PhDs from Stanford, Berkeley, Cornell, and industry veterans.
  • The company emphasizes a product‑first mindset: research must translate into usable tools for enterprises and eventually personal devices.
  • Hiring focus: performance engineers, research engineers, and infrastructure specialists who can build large‑scale, cost‑efficient AI pipelines.

8. Closing Thoughts

  • Efficiency ≠ cheapness: Doing more with less does not mean sacrificing intelligence; it expands the horizon of solvable problems.
  • Next paradigm: Moving from "scale‑up" (more tokens, larger models) to "scale‑down" (compact representations, continual learning) is the key to long‑term AI memory.
  • Call to action: Engram invites collaborators, investors, and talent to join their mission of building AI that truly remembers and learns.

Where to learn more: https://engram.com – contact Dan at dan@engram.com for partnership or hiring inquiries.

Sources