MemTensor/MemT-1.0

What it solves

Large language models typically lose earlier context unless it is in the current window, and handling long histories is costly because the full context must be stored. MemT-1.0 solves this by making memory a built-in part of the model, so it can retain and use past information without needing the original text at inference time. Its persistent state stays constant in size, cutting storage by about 66× compared with a full-context KV cache at 128K history.

How it works

MemT-1.0 adds a Native Memory Block beside the backbone attention. This block has two parts: a Local Memory Block that keeps a dynamic, multi-channel memory matrix and evolves over interaction steps, and a Hyper Memory Block with trainable memory query, key, value projections plus erase, write, forget, and output gates. During generation, memory attention reads from the current memory state, normalizes and gates the readout, and adds it to the backbone attention output. At the end of each turn or segment, a GDN-2 mechanism performs token-level memory updates. The project also includes a two-stage training pipeline supported by 34B tokens of data and dedicated infrastructure, and it demonstrates an experience-guided agentic coding framework that stores procedural experience in native memory.

Who it’s for

Researchers and developers working on memory-enhanced LLMs, especially those interested in long-horizon tasks, agentic systems, and applications that need to persist user-specific knowledge without storing full conversation logs. The quick start targets users with Linux servers, NVIDIA CUDA, and Python 3.11 or newer, and it uses SGLang for deployment and a Python client for conversations.

Highlights

• Two model sizes: MemT-1.0-Nano (36B total, 3B activated) and MemT-1.0-Flash (288B total, 17B activated), each with trainable memory parameters and a dynamic memory state.
• Achieves about 66× storage reduction versus a full-context KV cache at 128K history.
• Ranks first and second among the evaluated parametric-memory methods on 11 memory benchmarks.
• Provides an experience-guided agentic coding framework that improves task performance without adding experience text to the task context.
• Supports saving and resuming memory state as a snapshot, decoupling memory from the conversation transcript.
• Offers a non-commercial license for the repository software (PolyForm Noncommercial 1.0.0) and a separate CC-BY-NC-SA license for the paper.