grishahq/recursive-llm

Recursive Language Models for efficient long-context processing. Analyze 1M+ tokens by storing context in a Python REPL while reducing LLM token usage.

What it solves

Recursive Language Models (RLM) solves the problem of analyzing very large contexts that are either too expensive or too large to fit into a standard model prompt. Instead of sending the entire source text to the LLM, RLM allows the model to interact with the data through a restricted Python REPL, enabling it to search, filter, and process only the necessary sections of the text.

How it works

RLM stores the source text as a variable within a persistent Python REPL environment. The root model receives the query and instructions but not the full text. The model then generates Python code to explore the context (e.g., using regex or slicing), performs local computations, or makes recursive sub-calls to other LMs to process specific chunks of data. This process continues until the model produces a final answer. To ensure safety and cost control, the REPL runs in an isolated subprocess using RestrictedPython with hard timeouts and optional POSIX resource limits.

Who it’s for

RLM is designed for developers and researchers who need to process massive documents or datasets where direct completion is too costly or exceeds context windows, and where the task benefits from programmatic search, exact counting, or local computation.

Highlights

  • Externalized Context: Keeps large source texts in the REPL rather than repeating them in prompts.
  • Bounded Execution: Implements tree-wide limits on API calls, tokens, estimated cost, and elapsed time.
  • Provider Portability: Supports over 100 providers via LiteLLM, including OpenAI, Anthropic, DeepSeek, and local models via Ollama or llama.cpp.
  • Observable Runs: Provides detailed per-run statistics, typed failures, and complete execution trajectories in JSONL format.
  • Cost Optimization: Allows the use of a high-capability root model for decision-making and a cheaper model for recursive sub-calls.
  • Answer Validation: Supports custom deterministic validators to reject and provide feedback on final answers without restarting the run.

Related

  • Project
    avbiswas/fast-rlm

    A minimal implementation of Recursive Language Models (RLMs) that allows LLMs to handle arbitrarily long prompts by programmatically exploring data via a Python REPL and recursive sub-agents.

  • Project
    alexzhang13/rlm

    An inference engine and training environment for Recursive Language Models (RLMs) that enable LLMs to handle near-infinite context by programmatically decomposing tasks via recursive sub-calls in a REPL environment.

  • Project
    Trampoline-AI/predict-rlm

    A runtime for Recursive Language Models that allows LLMs to solve complex, adaptive workflows by writing and executing Python code in a sandboxed REPL.

  • Project
    simonw/llm

    A CLI tool and Python library for interacting with various remote and local Large Language Models, featuring SQLite logging and structured data extraction.

  • Project
    microsoft/LMOps

    A research initiative focused on fundamental technologies to improve LLM prompting, inference speed, and context handling for building AI products.