NX-AI/xlstm

xLSTM – Extended Long Short‑Term Memory

What it is – xLSTM is a new recurrent neural‑network architecture that builds on the classic LSTM. It introduces exponential gating, a matrix memory and several normalization/stabilisation tricks, allowing it to scale to very large language models (e.g., a 7 B‑parameter model) while keeping inference fast and memory‑efficient. The authors claim it can rival Transformers and state‑space models on language‑modeling benchmarks.

Key components

• xLSTMBlockStack – a drop‑in replacement for a stack of Transformer blocks; internally it mixes three block types (mLSTM, sLSTM, and a feed‑forward) and can be configured via dataclasses.
• xLSTMLMModel – a language‑model wrapper that adds token embeddings and an LM head on top of the block stack.
• mlstm_kernels – a separate package providing custom CUDA/Triton kernels ("sLSTM" kernels) that dramatically speed up the recurrent operations.
• xLSTMLarge – a single‑file implementation of the 7 B‑parameter model used in the follow‑up paper, with configurable kernels for different hardware.

Installation

# optional: create the exact conda environment the authors used
conda env create -f environment_pt240cu124.yaml
conda activate xlstm

# install the fast kernels (required for the 7B model)
pip install mlstm_kernels

# install the library itself
pip install xlstm   # or: git clone https://github.com/NX-AI/xlstm && pip install -e .

The package works with PyTorch ≥ 1.8. For GPU acceleration you need a recent NVIDIA GPU (CUDA compute capability ≥ 8.0) to use the Triton kernels; otherwise the pure‑PyTorch fallback works on any platform.

Quick start (7B inference)

import torch
from xlstm.xlstm_large.model import xLSTMLargeConfig, xLSTMLarge

cfg = xLSTMLargeConfig(
    embedding_dim=512,
    num_heads=4,
    num_blocks=6,
    vocab_size=2048,
    return_last_states=True,
    mode="inference",
    chunkwise_kernel="chunkwise--triton_xl_chunk",
    sequence_kernel="native_sequence__triton",
    step_kernel="triton",
)
model = xLSTMLarge(cfg).to("cuda")
inputs = torch.randint(0, 2048, (3, 256), device="cuda")
out = model(inputs)
print(out.shape)   # (3, 256, 2048)

A notebook (notebooks/xlstm_large/demo.ipynb) demonstrates the same workflow.

Hardware recommendations

• NVIDIA GPUs – best performance with the Triton kernels; the repo mentions successful runs on RTX 3080/3090 and newer (CC 8.0+).
• AMD GPUs – Triton kernels may still run, but the authors suggest falling back to native PyTorch kernels.
• Apple Silicon – use the community‑maintained xLSTM-metal port (MLX) for a Metal‑native implementation.

Models

• xLSTM‑Large 7B – a 7 billion‑parameter recurrent LLM trained on 2.3 T tokens; weights are hosted on Hugging Face (https://huggingface.co/NX-AI/xLSTM-7b).
• Smaller research models are available via the xLSTMBlockStack and xLSTMLMModel classes, which can be instantiated from YAML configs (the README shows examples).

Experiments The repository includes synthetic tasks (Parity, Multi‑Query Associative Recall) that illustrate the complementary strengths of the two sub‑blocks (sLSTM for state‑tracking, mLSTM for memorisation). Running them is as simple as:

PYTHONPATH=. python experiments/main.py --config experiments/parity_xlstm11.yaml

(Note: the training loops are minimal – no early stopping or test evaluation.)

Citation If you use the code or the 7 B model, cite the two papers listed in the README (NeurIPS 2024 xLSTM paper and ICML 2025 xLSTM‑7B paper).

---

All information is taken directly from the repository’s README; no external assumptions are added.