yifanzhang-pro/recurrent-looped-tranformer

Official Project Page for Recurrent Looped Transformer (RLT) (https://arxiv.org/abs/2610.07591)

What it solves

Recurrent Looped Transformer (RLT) addresses the limitation of standard Transformers in length generalization—the ability to perform tasks on sequences much longer than those seen during training. It specifically targets algorithmic tasks (like parity and state-tracking) where the computation path needs to grow with the sequence length to maintain accuracy.

How it works

RLT modifies the Transformer architecture to introduce recurrence across tokens:

  • Recurrent Feedback: The final decoder state of the previous token is fed back into the current token's decoder input via a gated merge. This creates a computation path that grows linearly with the sequence length.
  • Hybrid Memory: It uses a parallel causal encoder for global key-value (KV) memory and a decoder with sliding-window attention (SWA) for local memory.
  • State Continuity: The recurrent state continues seamlessly across the boundary between the prompt and the response.
  • Feedback Variants: The project explores different feedback intervals ($B$): RLT-1 updates every token, RLT-2 updates every chunk of $B$ tokens, and RLT-0 has no feedback.

Who it’s for

Researchers and developers working on sequence modeling, length generalization, and algorithmic reasoning in LLMs who want to move beyond the fixed-depth limitations of standard Transformers.

Highlights

  • Strong Length Generalization: Achieves 100% accuracy on parity and S5 swap tasks at lengths significantly exceeding training data (e.g., 256 bits for parity).
  • Recurrent Path: The computation path traverses $tL_D$ decoder blocks for $t$ tokens, while maintaining a fixed per-token cost.
  • Flexible Depth Allocation: Experiments demonstrate how splitting layers between the encoder and decoder affects performance on different algorithmic tasks.
  • Efficiency Trade-offs: RLT-2 (chunked feedback) provides a speed-up in training (up to 2.27×) compared to RLT-1 while maintaining high accuracy on some tasks.

Related

  • Dispatch
    GPT-6 Astra: Looped Transformers and the Debate Over Hidden Reasoning

    GPT-6 Astra introduces significant leaps in computer-use capabilities and likely employs looped transformer architectures to increase effective model depth without increasing parameter count.

  • Dispatch
    Transformer-based Encoder-Decoder Models Hugging Face Blog Post 2020

    Hugging Face’s 2020 blog post explains the transformer-based encoder-decoder architecture, detailing how it maps input sequences to variable-length outputs via encoder and decoder stacks, self-attention, cross-attention, and autoregressive generation, and shows how to use it with the 🤗Transformers library.

  • Dispatch
    Reformer: Pushing the Limits of Language Modeling with Memory-Efficient Transformers

    The Reformer model, introduced by Hugging Face in July 2020, enables training on sequences up to half a million tokens using less than 8 GB of RAM by combining LSH self‑attention, local self‑attention, chunked feed‑forward layers, reversible residuals, and axial positional encodings.

  • Project
    yifanzhang-pro/deep-delta-learning

    Deep Delta Learning (DDL) is a research implementation that adds a gated rank‑1 “edit” to the residual stream of Transformers. Each layer reads a direction, compares it to a learned target, and writes back a correction, optionally using expanded residual channels. The repo provides PyTorch (and Triton‑accelerated) GPT‑style models that can replace standard residual connections, showing modest improvements in validation loss and one‑shot accuracy on benchmark tasks.

  • Dispatch
    Hugging Face Reads: Long-range Transformers

    Hugging Face analyzes four key architectures—Longformer, Compressive Transformer, Linformer, and Performer—designed to reduce the quadratic memory and time complexity of standard Transformer self-attention to linear complexity.