Lightning-AI/lightning-thunder

PyTorch compiler that accelerates training and inference. Get built-in optimizations for performance, memory, parallelism, and easily write your own.

What it solves

Thunder is a source-to-source compiler for PyTorch designed to simplify the process of optimizing deep learning models for both training and inference. It bridges the gap between "unoptimized" PyTorch eager mode and highly optimized execution by providing a transparent way to apply performance enhancements without the opacity of traditional compilers.

How it works

Thunder operates in three primary stages:

  1. Acquisition: It interprets Python bytecode to convert a model into a straight-line Python program (its Intermediate Representation).
  2. Transformation: It applies a system of transforms to this IR to handle distribution, change numerical precision, or modify the computation graph.
  3. Execution: It routes the resulting trace to various executors, including fusion engines (like NVFuser or torch.compile), specialized libraries (cuDNN SDPA, TransformerEngine), custom Triton/CUDA kernels, or standard PyTorch eager operations.

Who it’s for

It is intended for developers and researchers who need to maximize the performance of PyTorch models, specifically those working with LLMs or other large-scale models on NVIDIA hardware (including Blackwell), and those who want an inspectable and extensible way to apply quantization, mixed precision, and distributed strategies.

Highlights

  • Performance Gains: Capable of running PyTorch models up to 40% faster.
  • Flexible Precision: Supports FP4, FP6, FP8, and mixed precision strategies.
  • Distributed Scaling: Built-in support for Tensor Parallelism (TP), Pipeline Parallelism (PP), and Data Parallelism (DP).
  • Inspectable IR: Uses a Pythonic IR that allows users to profile, map ops to kernels, and interactively inspect programs.
  • Composable Recipes: Optimizations can be bundled into recipes for easy porting across different model families.
  • CUDAGraphs Support: Includes plugins to reduce CPU overhead via CUDAGraphs.

Related

  • Project
    pytorch/TensorRT

    Torch-TensorRT integrates NVIDIA TensorRT into PyTorch to accelerate inference latency by up to 5x on NVIDIA GPUs.

  • Project
    Lightning-AI/pytorch-lightning

    A deep learning framework that abstracts away the engineering boilerplate of PyTorch, enabling researchers to pretrain and finetune AI models at scale across various hardware.

  • Project
    facebookincubator/AITemplate

    A Python framework that transforms deep neural networks into optimized CUDA/HIP C++ code for high-performance inference serving on NVIDIA and AMD GPUs.

  • Project
    mosaicml/composer

    A PyTorch-based deep learning library that simplifies distributed training at scale, providing tools for memory optimization and algorithmic speedups to reduce training time and cost.

  • Project
    NVIDIA/torch-harmonics

    A PyTorch library for differentiable signal processing on the sphere, enabling spherical harmonic transforms and neural operators for physics-based AI.