yifanzhang-pro/recurrent-looped-tranformer

What it solves

It addresses the limitation of fixed-depth computation in standard Transformers, where the amount of processing per token is constant regardless of the sequence length. The Recurrent Looped Transformer (RLT) aims to provide "infinite temporal depth," allowing the model to extend its computation path as the sequence grows, which is intended to improve latent reasoning capabilities.

How it works

RLT uses a hybrid architecture combining a causal encoder and a recurrent decoder:

• Causal Encoder: Builds a global key-value (KV) memory of the input.
• Recurrent Decoder: Combines this global memory with sliding-window attention (SWA) for local context and a feedback loop where the final hidden state of the previous token is fed back into the processing of the next token.
• Unified Execution: The model uses the same state transition logic across pretraining, supervised fine-tuning (SFT), and RL replay, ensuring consistency between how the model is trained and how it generates text.

Who it’s for

Researchers and developers working on transformer architectures, recurrent neural networks, and latent reasoning who want to explore alternatives to standard attention mechanisms for longer sequences.

Highlights

• Infinite Temporal Depth: The computation path grows linearly with the sequence length while keeping the number of blocks per token fixed.
• Model-Hardware Co-design: Optimized for efficient training and inference through parallel encoder work and memory reuse.
• Consistent State Transitions: Eliminates structural mismatches between prompt processing and response generation.
• State-Tracking Performance: Demonstrated superior accuracy on synthetic state-tracking tasks compared to standard Transformers and GRUs, especially when evaluating on sequences longer than those seen during training.