Training Decision Transformers for Offline Reinforcement Learning

Decision Transformers Shift RL to Sequence Modeling

Decision Transformers treat reinforcement learning (RL) as a conditional-sequence modeling problem rather than a traditional policy optimization task. Instead of fitting a value function to maximize cumulative rewards, the Decision Transformer uses a sequence modeling algorithm (a Transformer) to generate future actions based on a desired return, past states, and past actions.

This approach represents a paradigm shift in RL by utilizing generative trajectory modeling—modeling the joint distribution of states, actions, and rewards—to replace conventional RL algorithms. The model does not maximize return in the traditional sense; instead, it generates actions predicted to achieve a specific target return.

Model Architecture and Process

The Decision Transformer architecture follows these steps:

  1. Input Sequence: The model receives the last $K$ timesteps, each consisting of three components: Return-to-go, State, and Action.
  2. Embedding: Tokens are embedded using a linear layer for vector states or a CNN encoder for image frames.
  3. Prediction: A GPT-2 based model processes these inputs and predicts future actions via autoregressive modeling.

Implementing Offline Decision Transformers

Offline Decision Transformers are trained on datasets collected from other agents or human demonstrations without any direct interaction with the environment during training.

Dataset Preparation and Custom Data Collators

To train a Decision Transformer, raw RL data must be preprocessed into a format suitable for sequence modeling. Using the halfcheetah-expert-v2 dataset from the Hugging Face Hub, the following preprocessing steps are required:

  • Normalization: Each feature is normalized by subtracting the mean and dividing by the standard deviation.
  • Return Computation: Discounted returns are pre-computed for each trajectory.
  • Scaling: Rewards and returns are scaled by a factor of 1000.
  • Sampling Distribution: The sampling distribution is augmented to account for the length of expert agent trajectories.

These steps are implemented via a custom Data Collator, which returns batches containing states, actions, rewards, returns-to-go, timesteps, and attention masks.

Training with the Hugging Face Trainer

Training is performed using the Hugging Face Trainer class. Because the standard DecisionTransformerModel does not inherently return a loss value, a wrapper class (TrainableDT) is used to calculate the L-2 norm (mean squared error) between the model's action predictions and the target actions from the dataset.

Key Training Hyperparameters:

  • Epochs: 120
  • Batch Size: 64
  • Learning Rate: $1e-4$
  • Weight Decay: $1e-4$
  • Warmup Ratio: 0.1
  • Optimizer: AdamW
  • Max Gradient Norm: 0.25

Future Directions for RL in Transformers

Hugging Face intends to expand its support for the Deep Reinforcement Learning community through several planned initiatives:

  • Online Training: Expanding the repository to include Decision Transformer models trained or fine-tuned in an online setting.
  • Tool Integration: Integrating sample-factory version 2.0 into their ecosystem.

Sources