facebookresearch/fairseq2

What it solves

fairseq2 is a sequence modeling toolkit designed for researchers to train custom models for content generation tasks. It replaces the original fairseq framework with a more modular, extensible architecture that allows researchers to maintain their own project codebases without needing to fork the entire library.

How it works

It provides a clean API and a modular design philosophy. The toolkit integrates with modern PyTorch tooling, including torch.compile and PyTorch FSDP for high-performance training. It features a C++ based streaming data pipeline for high throughput, specifically supporting speech and video decoding.

Who it’s for

AI researchers focusing on sequence modeling and content generation, particularly those working with large-scale models (70B+ parameters) and diverse modalities like text, speech, and video.

Highlights

• Scalable Training: Supports multi-GPU and multi-node training using DDP, FSDP, and tensor parallelism for models over 70B parameters.
• Extensible Architecture: Uses a setuptools extension mechanism to register new models, optimizers, and schedulers without forking the library.
• Native vLLM Support: Includes native support for vLLM alongside built-in sampling and beam search generators.
• Integrated Recipes: Provides first-party recipes for instruction finetuning and preference optimization.
• Asset Management: Uses programmatic asset cards for version-controlled access to models, datasets, and tokenizers.