lucidrains/transfusion-pytorch

What it solves

This project implements a multi-modal model architecture that can handle both discrete text tokens and continuous modality data (like images or audio) within a single transformer. It solves the problem of integrating disparate data types—predicting the next token for text and using generative processes for continuous data—into one unified model.

How it works

The model combines autoregressive next-token prediction for text and flow matching (a substitute for the original paper's diffusion) for continuous modalities. It treats text as torch.long and continuous modalities as torch.float. The architecture supports multiple different modalities with specific latent dimensions and shapes, and can optionally integrate custom encoders and decoders to handle the raw data conversion.

Who it’s for

Researchers and developers building multi-modal AI models that need to generate interleaved text and non-text data (e.g., images, audio, or robotic actions) using a single transformer backbone.

Highlights

• Multi-Modality Support: Capable of handling any number of different continuous modalities alongside text.
• Flow Matching: Uses flow matching instead of standard diffusion for continuous data generation.
• Batched Sampling: Includes a sample_many function that uses KV-caching and joint ODE trajectories to decode multiple samples in parallel.
• Flexible Input: Supports interleaved sequences of text and modality tensors, as well as the ability to force a specific modality at the start of generation.
• Integrated Encoding: Optional support for providing custom modality encoders and decoders.