lucidrains/x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

What it solves

It provides a concise and fully-featured implementation of the Transformer architecture that allows researchers and developers to easily integrate a wide array of experimental features from various academic papers into their models without writing them from scratch.

How it works

The library offers flexible wrappers and modules (XTransformer, TransformerWrapper, Encoder, Decoder) that can be configured via simple keyword arguments. It supports various configurations including full encoder-decoder, decoder-only (GPT-like), encoder-only (BERT-like), and Vision Transformer (ViT) setups. It integrates advanced optimizations like Flash Attention for memory efficiency and speed, and supports multimodal tasks like image captioning and language-vision models (e.g., PaLI).

Who it’s for

AI researchers and machine learning engineers who want to experiment with different Transformer variants, normalization techniques, and attention mechanisms to improve model performance or efficiency.

Highlights

  • Diverse Architecture Support: Supports encoder-only, decoder-only, and full encoder-decoder configurations.
  • Experimental Attention Features: Includes persistent memory KV, memory tokens (register tokens), talking-heads attention, and sparse top-k attention.
  • Advanced Normalization: Implements RMSNorm, SimpleRMSNorm, ScaleNorm, and L2 normalized embeddings.
  • Efficiency Optimizations: Integrated support for Flash Attention and multi-query/grouped-query attention (one KV head).
  • Feedforward Enhancements: Supports GLU variants (GELU, Swish), ReLU², and bias-free feedforward layers.
  • Vision Integration: Includes ViTransformerWrapper for image classification and multimodal tasks.

Related

  • Project
    facebookresearch/xformers

    A toolbox of optimized, customizable building blocks for Transformer models that accelerates research by providing memory-efficient and high-performance CUDA kernels.

  • Project
    lucidrains/vit-pytorch

    A comprehensive PyTorch library implementing the original Vision Transformer (ViT) and dozens of its advanced variants for image classification.

  • Project
    kyegomez/Gemini

    An open-source implementation of a multimodal transformer model that natively processes text, images, and audio within a single architecture.

  • Project
    OpenNMT/CTranslate2

    A C++ and Python library for efficient Transformer model inference, using quantization and custom runtimes to accelerate execution and reduce memory usage on CPU and GPU.

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.