Perceiver IO: A Scalable, Fully-Attentional Model for Any Modality

Perceiver IO is a scalable neural network architecture that enables the use of the Transformer's self-attention mechanism across any modality—including text, images, audio, video, and point clouds—without the quadratic compute and memory costs typically associated with high-dimensional data. By performing the bulk of its computation in a small, fixed-size latent space, Perceiver IO removes the dependency between input size and the complexity of the self-attention layers.

Overcoming Transformer Scaling Limitations

Standard Transformer architectures scale poorly in compute and memory because their self-attention mechanism requires a pairwise dot product for all inputs in every layer. To handle high-dimensional data, existing models typically rely on modality-specific preprocessing to discretize data into tokens (e.g., ViT uses image patches, Wav2Vec2 uses feature encoders for audio).

Perceiver IO solves this by employing self-attention on a set of latent variables rather than on the inputs themselves. The inputs are used only for cross-attention with these latents. This ensures that the Transformer encoder's complexity depends linearly on the input size, while the latent attention remains independent of the input size. Perceiver IO further extends this by using a similar cross-attention mechanism for the output, allowing the model to handle arbitrary output shapes.

Core Architecture and Workflow

All Perceiver variants are built upon the PerceiverModel class, which utilizes three optional components to handle different data types:

  1. Preprocessor: Embeds raw inputs (text, images, etc.) into vectors.
  2. Decoder: Decodes the final hidden states of the latents into a useful format, such as classification logits or optical flow.
  3. Postprocessor: Converts decoder outputs into specific features, primarily used in auto-encoding tasks.

The Processing Pipeline

  • Input Stage: Inputs are optionally processed by a preprocessor.
  • Cross-Attention: The latent variables produce queries (Q), while the preprocessed inputs produce keys and values (KV). This maps the high-dimensional input into the latent space.
  • Latent Processing: A repeatable block of self-attention layers updates the latent embeddings. The output is a tensor of "last hidden states" of the latents.
  • Output Stage: An optional decoder performs another cross-attention operation where trainable embeddings produce queries (Q) and the latents produce keys and values (KV). A postprocessor may then refine these outputs.

Modality-Specific Implementations

Text Classification

Perceiver IO can process raw UTF-8 bytes directly, removing the need for explicit tokenization (like WordPiece or BPE). For text classification, the PerceiverTextPreprocessor embeds byte IDs and adds absolute position embeddings. A PerceiverClassificationDecoder then uses cross-attention to map the latents to classification logits. In masked language modeling (PerceiverForMaskedLM), a PerceiverBasicDecoder maps latents back to the input sequence length to predict missing bytes.

Image Classification

For vision tasks, the model uses a PerceiverImagePreprocessor. The authors tested three preprocessing methods:

  • Flattening pixels with a 1x1 convolutional layer and learned 1D position embeddings.
  • Flattening pixels with fixed 2D Fourier position embeddings.
  • Using a 2D convolutional + maxpool layer with fixed 2D Fourier position embeddings.

After pre-training on JFT, the PerceiverForImageClassificationConvProcessing variant achieved 84.5% top-1 accuracy on ImageNet. Notably, the version using only 1D learned position embeddings achieved 72.7% accuracy, demonstrating thes model's ability to perform without explicit 2D structural knowledge.

Optical Flow Estimation

Perceiver IO can estimate 2D pixel displacement between two image frames. The PerceiverForOpticalFlow model extracts 3x3 patches around each pixel, concatenates frames along the channel dimension, and flattens the spatial dimensions. The PerceiverOpticalFlowDecoder then uses the preprocessed inputs as queries to decode the latents back into a predicted flow map. This approach achieved state-of-the-art results on Sintel and KITTI benchmarks when trained on the AutoFlow dataset.

Multimodal Autoencoding

Using the PerceiverForMultimodalAutoencoding model, the architecture can learn joint distributions across multiple modalities. For example, on the Kinetics-700 dataset, it processes video frames, audio samples, and class labels simultaneously:

  • Images: Transformed via "space to depth" and concatenated with 2D Fourier embeddings.
  • Audio: Concatenated with fixed Fourier position embeddings.
  • Labels: Processed via a one-hot preprocessor.

These modalities are padded to a uniform channel dimension and concatenated. The decoder then auto-encodes the data in chunks to reconstruct the original modalities. The model achieved 45% top-1 accuracy for video classification while maintaining a 20.7 PSNR for video reconstruction.

Broader Applications and Versatility

Beyond the primary modalities, Perceiver IO has demonstrated success in other complex domains:

  • 3D Point Clouds: Trained on ModelNet40, the model achieved 85.7% top-1 accuracy, competing with specialized models like PointNet++.
  • Reinforcement Learning: In StarCraft II, the Perceiver replaced the original Transformer in AlphaStar, maintaining an 87% win-rate against the Elite bot without additional parameter tuning.

Because the architecture is task-agnostic, it can be adapted to any problem by designing new preprocessors and decoders, potentially unifying various modalities into a shared latent space.

Sources