lucidrains/x-transformers

What is x‑transformers?

x‑transformers is a lightweight, pure‑Python library that lets you build and experiment with many flavours of transformer models in PyTorch. It provides a single, flexible building block (XTransformer or TransformerWrapper) that can be configured as:

• Encoder‑decoder (like the original “Attention is All You Need” model)
• Decoder‑only (GPT‑style language models)
• Encoder‑only (BERT‑style models)
• Vision transformers for image classification or image‑to‑text tasks

The library also bundles a collection of research‑grade tweaks – flash‑attention, various normalisation schemes, memory tokens, sparse/top‑k attention, talking‑heads, GLU feed‑forwards, etc. – that can be turned on with a single flag.

---

Who might use it?

• Researchers who want a quick prototype of a new transformer variant without writing boiler‑plate code.
• Practitioners looking for a compact, configurable transformer for small‑to‑medium experiments (e.g., custom language models, image captioning, multimodal pipelines).
• Students learning how different architectural choices affect performance, since most options are exposed as simple arguments.

---

Core features (as described in the README)

Feature	What it does	How to enable
Flash Attention	Uses PyTorch 2.0’s scaled_dot_product_attention for linear‑memory, faster attention.	attn_flash=True
Persistent Memory KV	Adds learned key/value slots before normal attention.	attn_num_mem_kv=16
Memory Tokens	Learns extra tokens that travel through every layer (helps stability).	num_memory_tokens=20
ScaleNorm / RMSNorm / Simple RMSNorm	Alternative normalisation layers that can speed up convergence.	use_scalenorm=True, use_rmsnorm=True, use_simple_rmsnorm=True
GLU / Swish‑GLU / Feed‑forward without bias	Variants of the feed‑forward block shown to improve accuracy or throughput.	ff_glu=True, ff_swish=True, ff_no_bias=True
ReLU² activation	A simple activation that sometimes beats GELU.	ff_relu_squared=True
Sparse / Top‑k attention	Keeps only the top‑k attention scores, reducing compute.	attn_sparse_topk=8 (or attn_hard=True for argmax)
Talking‑heads	Linear mixes of heads before/after softmax.	attn_pre_talking_heads=True, attn_post_talking_heads=True
One‑KV‑head / grouped KV heads	Reduces the number of key/value heads for memory‑efficient decoding.	attn_one_kv_head=True or attn_kv_heads=2
Dropout options	Embedding dropout, stochastic depth, attention dropout, feed‑forward dropout.	emb_dropout, layer_dropout, attn_dropout, ff_dropout

---

Example usage snippets (from the README)

# Encoder‑decoder (seq2seq) model
from x_transformers import XTransformer
model = XTransformer(
    dim=512, enc_num_tokens=256, enc_depth=6, enc_heads=8,
    dec_num_tokens=256, dec_depth=6, dec_heads=8,
    tie_token_emb=True)

# GPT‑style decoder only
from x_transformers import TransformerWrapper, Decoder
model = TransformerWrapper(
    num_tokens=20000, max_seq_len=1024,
    attn_layers=Decoder(dim=512, depth=12, heads=8))

# Vision transformer for image classification
from x_transformers import ViTransformerWrapper, Encoder
model = ViTransformerWrapper(
    image_size=256, patch_size=32, num_classes=1000,
    attn_layers=Encoder(dim=512, depth=6, heads=8))

---

Installation

pip install x-transformers

---

TL;DR

x‑transformers is a modular, research‑focused PyTorch library that lets you spin up classic and cutting‑edge transformer architectures (language, vision, multimodal) with a handful of arguments, and experiment with a wide range of attention and normalization tricks from recent papers.