OFASys: A Framework for Multimodal Multitask Learning

Qwen has introduced OFASys, an AI framework designed to streamline multimodal multitask learning. By utilizing a simple "Instruction" interface, the system allows developers to build multimodal multitask learning jobs with a single line of code, removing the complexity of data preprocessing, model building, and training stability.

Simplified Task Definition via Instructions

OFASys replaces complex implementation processes with a template-based Instruction system. An Instruction defines the task input and desired output using a specific syntax (e.g., input -> output), allowing the system to automatically handle the underlying data pipeline.

  • Image Captioning Example: An instruction specifying [IMAGE:img] -> cap tells the system that there is an image input bound to the img column of the dataset and that the output should be the text sequence in the cap column.
  • Natural Language Inference (NLI) Example: For NLI tasks, the system supports multiple inputs for the encoder and specific signals like no_loss to avoid loss computation on repeated inputs, as well as closed_set to indicate a closed label set.

System Architecture and Design

OFASys uses a modular design to translate Instructions into executable task plans. This architecture ensures that different modalities can be processed through consistent inner interfaces.

Task Plan Components

Each task plan consists of a model hierarchy that includes:

  • Modality-Specific Components: Preprocessors, postprocessors, and adapters that handle the intricacies of specific data types.
  • Universal Model: A modality-agnostic computation model that fuses multimodal inputs and generates outputs. Because inputs and outputs are treated as representation sequences, the universal model remains versatile regardless of the modality.

Training and Execution

To manage the scale of multitask learning, OFASys implements the following:

  • Parameter Sharing: By default, the system shares trainable parameters between adapters and the universal model to maximize optimization across examples.
  • Task and Logical Schedulers: A task scheduler manages joint optimization and task precedence, while a logical scheduler handles the workflow across multiple physical devices.
  • Stage-wise Components: Out-of-the-box generators and criteria are provided to support training and inference.

Performance Validation: The OFA+ Model

To demonstrate the framework's effectiveness, Qwen developed OFA+, a generalist model capable of handling text, image, speech, video, and motion data simultaneously.

The researchers compared three versions of the model:

  1. OFA+ (Generalist): A generalist model based on OFA.
  2. OFA+ (Generalist MoE): An improved version utilizing modality-level Mixture of Experts (MoE).
  3. OFA+ (Specialist): The original OFA model fine-tuned on specific tasks for baseline comparison.

Results indicate that OFA+ preserves over 95% of the performance of specialist models while scaling to 23 diverse tasks across 7 different modalities. This suggests that multitask learning can provide generalist models with broad capabilities without sacrificing top-level performance on specific tasks.

Sources