fudan-generative-vision/Remory

What is Remory?

Remory is a plug‑in that adds a learned “residual memory” to large language models (LLMs) so they can compact long conversation histories without losing important information. When an agent (e.g., a coding assistant called Codex) needs to truncate its context, Remory first creates a normal summary, then encodes the remaining hidden states into a small set of “soft tokens”. Those tokens are stored alongside the summary and are automatically re‑inserted on the next generation step, letting the frozen base model continue as if it still had the full context.

Key ideas

Context compaction – Large agents quickly hit their token limits (256 K for Qwen 3.8‑27B). Remory keeps a concise summary plus a learned residual that captures what the summary missed.
Frozen base model – The underlying LLM (Qwen or GLM) is never fine‑tuned; only a lightweight memory network is trained, making the approach cheap to adapt to new models.
Online training – A command‑line tool (/remory train) can keep improving the memory network while you interact with a live Qwen session.
Zero‑code integration – Remory ships a SGLang hook and a small client library; you just launch the server and point your existing agent (or the provided r‑codex UI) at it.

Quick start (Qwen 3.8‑27B)

# 1️⃣ Clone & install
git clone https://github.com/1ring2rta/Remory.git && cd Remory
python deploy/install.py                     # creates .remory/venv with SGLang, CUDA tools
source .remory/venv/bin/activate

# 2️⃣ Launch the model + memory server (needs 2 × 80 GB GPUs)
CUDA_VISIBLE_DEVICES=0,1 python -m remory.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --remory-checkpoint mocoV3/Qwen3.8-27B-REMORY-1.9B \
  --tp 2 \
  --host 127.0.0.1 --port 8421

# 3️⃣ Open the Codex UI that talks to the server
source .remory/venv/bin/activate
r-codex -C /path/to/your/project   # use /compact to trigger compaction

The UI lets you run a normal chat, then call /compact. The system will:

Generate a summary of the removed messages.
Encode the residual hidden states into soft tokens.
Store the checkpoint (saved in .remory/memory.sqlite).
Continue the conversation from that checkpoint.

How to use it programmatically

from remory.adapters import Harness
from remory.client import Client

with Client("http://127.0.0.1:8421", session_id="demo") as client:
    agent = Harness(client, codec)   # codec = your chat template
    checkpoint = agent.compact(
        prefix_messages=system_and_tools,   # prompts that stay
        removed_messages=history,           # messages to discard
        summary_prompt=summary_prompt,
        summary_schema=summary_schema,
    )
    # generate a new answer using the saved residual memory
    answer = agent.generate(
        checkpoint=checkpoint,
        prefix_messages=system_and_tools,
        messages=new_user_messages,
    )

The Harness.compact method does the three‑step process (summary → residual → checkpoint) and returns a checkpoint that can be passed back to generate.

Training your own memory network

Remory ships two‑stage training scripts for the Qwen and GLM checkpoints:

Stage I – Reconstruction – learns to reconstruct hidden states from a short prefix.
Stage II – Summary‑conditioned residual – trains the residual memory to be useful when conditioned on the summary. Both stages keep the base LLM frozen; only the memory network (≈1‑2 B parameters) is updated. The training/ folder contains:

launch scripts for 4‑GPU runs
example JSONL data formats (history, summary, continuation)
resume / checkpoint handling

Benchmarks & results

The authors evaluated Remory on two long‑horizon agent suites:

BrowseComp (web‑search tasks) and Terminal‑Bench 2.1 (shell interaction).
Adding residual memory improved success rates and reduced tool‑output duplication.
On the SummHay benchmark, Remory achieved near‑full‑context performance while using only 5.2 % of the original token budget.

Performance numbers are visualised in assets/benchmarks.png and detailed in the accompanying paper (arXiv 2610.11287).

Where to get the models

Base model	Remory checkpoint	How to run
Qwen‑3.8‑27B	`mocoV3/Qwen3.8-27B-REMORY-1.9B`	SGLang or 🤗 Transformers
GLM‑5.3‑Flash	`mocoV3/GLM-5.3-Flash-REMORY-1.24B`	SGLang

Both checkpoints are hosted on Hugging Face and can be loaded with the --remory-checkpoint flag.

License & acknowledgements

MIT License (see LICENSE).
Builds on the DFlash and SpecForge memory ideas, and uses SGLang for high‑throughput inference.
The research paper is cited as:

@misc{xia2026remory,
  title  = {Remory: Learning Residual Memory for Context Compaction},
  author = {Xia, Hanchen and Chen, Baoyou and Ge, Yutang and Deng, Naihao and
            Yang, Senqiao and Dong, Zilong and Yuan, Weihao and Zhu, Siyu},
  year   = {2026},
  eprint = {2610.11287},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL}
}

TL;DR

Remory lets you keep a frozen LLM (like Qwen 3.8‑27B) running on long‑running agents by adding a tiny learned memory that sits after a normal summary. Install with the provided script, launch via SGLang, and either use the r‑codex UI or the Python Harness class to compact and continue conversations. Training is optional but supported for custom domains.