linkedin/Liger-Kernel

Efficient Triton Kernels for LLM Training

What it solves

Liger Kernel addresses the memory and throughput bottlenecks associated with training Large Language Models (LLMs). It reduces memory usage by up to 60% and increases multi-GPU training throughput by approximately 20%, enabling the use of larger batch sizes, longer context lengths, and larger vocabularies without running out of memory (OOM).

How it works

The project provides a collection of optimized Triton kernels that replace standard PyTorch layers. It uses techniques such as kernel fusion, in-place replacement, and chunking to optimize the forward and backward passes of critical LLM components. It implements Hugging Face compatible versions of RMSNorm, RoPE, SwiGLU, and CrossEntropy (including FusedLinearCrossEntropy). It also provides specialized kernels for post-training alignment and distillation tasks (e.g., DPO, ORPO, CPO, SimPO, KTO).

Who it’s for

It is designed for AI researchers and engineers training or fine-tuning LLMs who want to maximize hardware efficiency on NVIDIA (CUDA), AMD (ROCm), or Ascend NPU hardware.

Highlights

  • High Efficiency: Increases throughput by 20% and reduces memory usage by 60% for general training, with up to 80% memory savings for post-training kernels.
  • Ease of Use: Can be integrated into Hugging Face models with a single line of code or used as standalone modules.
  • Broad Compatibility: Works out of the box with Flash Attention, PyTorch FSDP, Microsoft DeepSpeed, and various trainer frameworks like Axolotl and LLaMA-Factory.
  • Exact Computation: Ensures mathematical accuracy with no approximations, verified via rigorous unit tests and convergence testing.
  • Hardware Support: Supports CUDA, ROCm, and Ascend NPU, with optional high-performance backends like cuTile and CuTe DSL for Hopper and Blackwell architectures.

Related

  • Project
    ServiceNow/Fast-LLM

    A high-performance library for training large language models that uses 3D parallelism and optimized kernels to reduce training time and cost.

  • Project
    unslothai/unsloth-zoo

    A collection of utilities and optimized kernels for Unsloth that enables faster and more memory-efficient fine-tuning of LLMs and multimodal models with significantly reduced VRAM usage.

  • Project
    NX-AI/xlstm

    xLSTM is a novel recurrent neural‑network architecture that extends LSTM with exponential gating and a matrix memory, enabling large‑scale language models (e.g., a 7 B‑parameter model). The repo provides PyTorch code, custom CUDA/Triton kernels (via the `mlstm_kernels` package), a single‑file 7B implementation, configuration utilities, and pretrained weights on Hugging Face. Installation is via conda + pip, and the library works on NVIDIA GPUs (fast Triton kernels) with fallbacks for AMD or Apple Silicon.

  • Project
    flagos-ai/FlagGems

    A high-performance operator library implemented in Triton that enables LLM training and inference to run across diverse AI accelerators using PyTorch APIs.

  • Dispatch
    Fine-tuning LLMs to 1.58-bit: Extreme Quantization with BitNet

    Hugging Face demonstrates that existing LLMs, such as Llama 3 8B, can be fine-tuned to 1.58-bit ternary precision using a dynamic warmup quantization strategy, significantly reducing memory and energy requirements while maintaining strong performance.