lucidrains/transfusion-pytorch

What it solves

Transfusion-pytorch provides a PyTorch implementation of the Transfusion architecture, which allows a single multi-modal model to both predict the next text token (autoregressive) and generate images or other modalities using diffusion-like processes. It solves the problem of having separate models for different modalities, unifying them into one transformer-based framework.

How it works

The project implements a unified transformer that handles different data types based on their tensor type: torch.long for text tokens and torch.float for continuous modalities. It replaces the original paper's diffusion process with flow matching to improve generation quality. The model can be configured with custom encoders and decoders to handle the conversion between raw modality data (like images) and the latent space the transformer operates in.

Who it’s for

Researchers and developers building multi-modal AI models that need to generate interleaved text and continuous data (such as images, audio, or robot actions) within a single model.

Highlights

• Unified Architecture: Combines next-token prediction for text and flow matching for continuous modalities in one model.
• Multi-Modality Support: Extends beyond images to support any number of different modalities with distinct latent dimensions.
• Efficient Sampling: Includes sample_many for parallel batch decoding of interleaved text and modalities using KV-caching and joint ODE trajectories.
• Customizable: Supports the integration of custom modality encoders and decoders.
• Flexible Generation: Allows forcing a specific modality at the start of generation to skip text decoding.