Is One Layer Enough? Training a Single Transformer Layer for RL Post-Training

Is One Layer Enough? Training a Single Transformer Layer for RL Post-Training

Summary of Findings

Training a single transformer layer can recover most of the performance gains achieved by full-parameter reinforcement learning (RL) post-training in large language models (LLMs). In some instances, RL gains are so concentrated in specific layers, and training a single layer in isolation can actually surpass the results of updating all parameters uniformly.

The Layer Contribution Metric

To quantify the impact of individual layers, researchers introduced the concept of "layer contribution," which measures the fraction of the total improvement gained through full RL training that is recovered when only a single layer is trained in isolation.

This metric allows for a systematic study of how RL adaptation is distributed across the transformer architecture, challenging the assumption that every layer contributes equally to the post-training process.

Distribution of RL Gains Across Layers

Across seven different models from the Qwen3 and Qwen2.5 families, the study found a remarkably stable pattern of layer contribution:

  • Middle-Layer Concentration: RL gains are highly concentrated in a small subset of layers, typically located in the middle of the transformer stack.
  • Low Contribution at Extremes: Layers near the input and output ends of the model contribute substantially less to the overall RL improvement.
  • Consistency Across Variables: This structural pattern remains strongly correlated across different datasets, tasks (including mathematical reasoning, code generation, and agentic decision-making), model families, and RL algorithms (including GRPO, GiGPO, and Dr. GRPO).

Experimental Setup and Scope

The study's findings are based on a systematic layer-wise study of RL training across a wide range of parameters:

  • Models: Qwen3 and Qwen2.5 families.
  • Algorithms: GRPO, GiGPO, and Dr. GRPO.
  • Task Domains: Mathematical reasoning, code generation, and agentic decision-making.
  • Subjects: Machine Learning (cs.LG) and Computation and Language (cs.CL).

Sources