facebookresearch/fairseq2

What it solves

fairseq2 is a sequence modeling toolkit designed for researchers to train custom models for content generation tasks. It replaces the original fairseq with a clean, modular API and an extensible architecture, allowing researchers to maintain their own project codebases independently rather than relying on a monolithic framework.

How it works

It leverages modern PyTorch tooling, including torch.compile and PyTorch FSDP, to provide high-performance training and inference. The toolkit includes a streaming-based data pipeline API written in C++ for high throughput, specifically supporting speech and video decoding. It also provides a structured API for deterministic configuration and programmatic asset cards for version-controlled access to models, datasets, and tokenizers.

Who it’s for

AI researchers and developers working on sequence modeling and content generation, particularly those needing to scale to 70B+ parameter models across multi-GPU and multi-node environments.

Highlights

• Scalable Training: Supports multi-GPU and multi-node training using DDP, FSDP, and tensor parallelism for very large models.
• Extensible Architecture: Uses a setuptools extension mechanism to register new models, optimizers, and trainers without needing to fork the library.
• litre-Integrated Inference: Native support for vLLM and built-in sampling and beam search generators.
• Specialized Recipes: First-party recipes for instruction finetuning and preference optimization.
• High-Throughput Data: C++ based streaming data pipeline for efficient handling of speech and video data.