OFA: Towards Building a One-For-All Model

OFA (One-For-All) is a unified multimodal pretrained model designed to integrate understanding and generation tasks across different modalities into a single framework. By utilizing instruction-based multitask pretraining, OFA aims to create a generalist model capable of processing diverse inputs and outputs without requiring architecture changes for different tasks.

Unified Framework for Modalities, Architecture, and Tasks

OFA achieves generalist capabilities by focusing on three core pillars of unification: modality agnosticism, task agnosticism, and task comprehensiveness.

Unification of Modalities

To handle diverse inputs, OFA discretizes non-text modalities into tokens. Images are represented using VQ (Vector Quantization) tokens, and bounding boxes are discretized using bins, following the pix2seq approach. This allows the model to treat all modalities as sequences of tokens.

Unification of Architecture

OFA employs a universal Transformer encoder-decoder architecture, similar to the T5 model. To process image inputs, the model uses the first three blocks of ResNet. For enhanced training stability and transfer performance, the architecture incorporates Normformer.

Unification of Tasks

OFA is pretrained using an instruction-based multitask learning approach. The model is trained on eight distinct tasks:

  • Vision-Language Tasks (5): Visual grounding, grounded captioning, visual question answering (VQA), image-text matching, and image captioning.
  • Vision Tasks (2): Detection and image infilling.
  • Language Task (1): Text infilling.

Instructions are inserted as text descriptions to help the model differentiate between these tasks, enabling the potential for zero-shot generation on unseen tasks based on new instructions.

Model Variants and Scale

OFA is released in five different sizes to support various deployment and research needs:

  • OFA-Tiny: 33M parameters
  • OFA-Medium: 93M parameters
  • OFA-Base: 180M parameters
  • OFA-Large: 470M parameters
  • OFA-Huge: 930M parameters

Performance and Experimental Results

OFA demonstrates competitive performance across multimodal and unimodal benchmarks, often matching or exceeding specialized models.

Multimodal Capabilities

  • Vision-Language Understanding: The OFA-Huge model achieves performance comparable to Flamingo (80B parameters) and CoCa (2B parameters) on VQA and SNLI-VE, and reaches the best performance on visual entailment.
  • Vision-Language Generation: OFA achieves state-of-the-art (SoTA) performance in image captioning under both cross-entropy and CIDEr optimization. In visual grounding, the base-size model outperforms previous SoTAs, with performance scaling consistently with model size.
  • Text-to-Image Generation: Due to its pretraining on image infilling, OFA can generate image codes and achieve a low FID score, which is further improved with finetuning on larger datasets.

Unimodal Capabilities

  • Natural Language Understanding (NLU): On the GLUE benchmark, OFA is competitive with RoBERTa and DeBERTa.
  • Natural Language Generation (NLG): OFA shows strong performance on Gigaword summarization, outperforming previous multimodal pretrained models.
  • Vision Understanding: On ImageNet classification, OFA achieves performance similar to self-supervised vision models like BeiT and MAE.

Compositional Generalization and Transfer

OFA exhibits the ability to transfer its learned capabilities to unseen tasks and domains through compositional generalization.

  • Unseen Tasks: The researchers introduced "Grounded VQA," a new task combining VQA and grounded captioning. By simply changing the instruction, OFA can leverage its existing capabilities to perform this new task.
  • Unseen Domains: OFA effectively transfers to unseen domains, such as performing visual grounding on animation images, by combining its pretraining on anime data and general-domain visual grounding.

Conclusion

The OFA model demonstrates that a single Transformer-based framework can unify tasks and modalities, providing a promising path toward the development of generalist foundation models in multimodal representation learning.

Sources