Context Language Models (CLMs)

What it is – A research codebase that implements Context Language Models, a new class of language models that treat their context as a mutable file. The model can read, write, and update this file during inference, learning what information to keep or discard. The approach works zero‑shot, with in‑context learning, and with reinforcement‑learning fine‑tuning.

Why it matters – Traditional LLMs have a fixed‑size context window; when inputs exceed that window they must truncate or off‑load information, which is often sub‑optimal. CLMs let the model decide dynamically what to retain, leading to higher accuracy and lower compute on a range of benchmark tasks (e.g., BrowseComp‑Plus, EdgeBench, multi‑repo agent‑swarm).

Key components (as listed in the repo)

Component	Purpose
`clm/clm_harness`	Core CLM implementation built on the Harbor agent framework.
`clm/clm_icl`	Code for in‑context learning – steering CLMs with natural‑language instructions via a skill‑optimization loop.
`clm/clm_rl`	Reinforcement‑learning pipeline that fine‑tunes CLMs (e.g., Qwen3.5‑9B) to improve performance on context‑management tasks.
`clm/clm_contextbench`	Integration with ContextBench, a benchmark suite that measures how agents manage context overflow.
`suffix_cache_reuse/`	Techniques for efficient serving of CLMs by re‑using suffix caches; includes adapters for SGLang and vLLM.

Getting started (quick‑start)

# Install the repository in editable mode
pip install -e .

# Run a minimal CLM agent on any Harbor task
clm-harbor run -p <harbor-task> -a clm-minimal -m openai/<model> \
  --agent-kwarg api_base=http://localhost:8000/v1

clm-harbor is the Harbor command‑line interface with a built‑in CLM agent (-a clm-minimal).
Replace <harbor-task> with the name of a Harbor benchmark (e.g., browsecomp_plus).
Replace <model> with the identifier of the underlying LLM you want to wrap (e.g., gpt-4o).

Example: Running on ContextBench

harbor jobs start --repo RulinShao/ContextBench -d contextbench \
  -a clm_contextbench:ClmStream -m openai/<model> \
  --agent-kwarg api_base=http://localhost:8000/v1

This single command evaluates a CLM on every task in the ContextBench suite.

Community integrations

Pi – install a CLM‑enabled version of the Pi agent: pi install npm:@lolipopshock/pi-clm.
OpenCode – a community‑maintained plugin (opencode-clm) lets OpenCode users run CLMs.

When to use CLMs

Long‑running agents that need to remember and update information over many interactions (e.g., autonomous web‑browsing, multi‑repo code assistants).
Compute‑constrained environments where reducing FLOPs while preserving performance is critical.
Research on context‑management strategies, in‑context learning, or RL‑based LLM fine‑tuning.

Citation

If you use the code or the ideas in a publication, cite the accompanying arXiv paper:

@article{shao2026context,
  title   = {Context Language Models},
  author  = {Shao, Rulin and Shen, Shannon Zejiang and Yin, Junjie Oscar and Li, Yuetai and Wang, Minheng and Ivison, Hamish and Poovendran, Radha and Lambert, Nathan and Xiao, Teng and Lewis, Mike and Yih, Wen-tau and Zettlemoyer, Luke and Koh, Pang Wei},
  journal = {arXiv preprint arXiv:2609.37725},
  year    = {2026}
}

License

The project is released under CC BY‑NC 4.0 (non‑commercial Creative Commons). See LICENSE and NOTICE for details.

facebookresearch/context-language-models