Jack of All Trades (JAT) Multi-Purpose Transformer Agent

Hugging Face has announced Jack of All Trades (JAT), a generalist agent project designed to perform a wide variety of vision-and-language and decision-making tasks using a single network. This project serves as an open reproduction of the Gato architecture, providing a comprehensive set of expert RL agents, a specialized training dataset, and a multi-modal transformer model.

JAT Dataset and Expert Policies

The JAT project provides the first dataset specifically designed for generalist agent training, containing hundreds of thousands of expert trajectories. These trajectories were generated by training state-of-the-art expert agents across several diverse environments, including:

  • Atari: Classic arcade games.
  • BabyAI: Simple navigation environments (utilizing the BabyAI bot).
  • Meta-World: Robotic manipulation tasks.
  • MuJoCo: Physics-based continuous control.

To enable a unified user interface, the JAT dataset also incorporates textual data from Wikipedia, Oscar, OK-VQA, and Conceptual-Captions.

JAT Agent Architecture

JAT is built on EleutherAI's GPT-Neo implementation. The architecture is designed to handle sequential decision tasks by interleaving observation embeddings, action embeddings, and corresponding rewards.

Embedding and Encoding Mechanisms

JAT uses different encoding strategies based on the data modality:

  • Images (e.g., Atari): Processed via a Convolutional Neural Network (CNN).
  • Continuous Vectors: Processed via a linear layer.
  • Discrete Values: Processed via a linear projection layer.
  • Text: Tokenized using the GPT-2 strategy.
  • General Images: Processed using a ViT-type encoder.

Prediction and Loss Functions

The model predicts the next embedding autoregressively using a causal mask, shifting observations by one time step so the agent must predict the next action based on previous observations and actions. The loss is computed separately for each modality:

  • MSE Loss: Used for images and continuous values.
  • Cross-Entropy Loss: Used for discrete values.

The final loss is the average of the losses for each element in the sequence.

Performance Results

JAT was evaluated across 157 training tasks, achieving an average performance of 65.8% compared to the expert agents across four primary domains.

Domain-Specific Performance

  • BabyAI: 99.0% of the expert's score, failing to exceed 50% of the expert on only one task.
  • MuJoCo: 84.8% of the expert's score.
  • Meta-World: 65.5% of the expert's score.
  • Atari 57: 14.1% of the expert's score (equivalent to 37.6% of human performance), exceeding human performance in 21 games.

While the model shows rudimentary capabilities in NLP and CV tasks, its primary strength lies in its ability to use a single network to mimic expert performance across these diverse RL domains.

The Impact of Observation Prediction

Research conducted during the JAT project explored whether asking an agent to predict future observations (in addition to maximizing rewards) improves learning. By using a weighting parameter $\kappa$ to balance observation loss and action loss, the team discovered a "sweet spot" at $\kappa = 0.005$.

At this specific weight, learning to predict observations improved the agent's learning efficiency. However, when $\kappa$ was too high (e.g., 0.5), the objective of predicting observations hindered the learning process. This suggests that auxiliary objectives can be beneficial to generalist agents if balanced correctly.

Future Research Directions

To further evolve the JAT agent, Hugging Face identifies three primary areas for improvement:

  1. Data Quality: Expanding the dataset by collecting more trajectories from a wider variety of expert agents to reduce bias.
  2. Offline RL: Moving beyond basic Behavioral Cloning to allow the agent to utilize sub-optimal trajectories and potentially outperform the expert.
  3. Sampling Strategies: Implementing a dynamic multi-task sampling strategy to focus on more challenging tasks rather than uniform sampling.

Sources