linkedin/Liger-Kernel

What it solves

Liger Kernel addresses the high memory consumption and computational overhead associated with training Large Language Models (LLMs). It specifically targets bottlenecks in common layers and loss functions, allowing for longer context lengths, larger batch sizes, and more massive vocabularies without triggering out-of-memory (OOM) errors.

How it works

Liger Kernel provides a collection of optimized Triton kernels that replace standard PyTorch implementations. It uses techniques such as kernel fusion, in-place replacement, and chunking to reduce memory footprints and increase throughput. It offers several ways to integrate:

• Auto-patching: A wrapper class (AutoLigerKernelForCausalLM) that automatically applies optimizations to supported models.
• Model-specific APIs: Functions to monkey-patch Hugging Face models (e.g., apply_liger_kernel_to_llama).
• Custom Composition: Individual kernels that can be used as standalone modules in a custom model architecture.

It supports multiple backends including Triton, cuTile, and CuTe DSL (for NVIDIA Hopper and Blackwell architectures).

Who it’s for

It is designed for AI researchers and engineers training or fine-tuning LLMs, particularly those using the Hugging Face ecosystem, PyTorch FSDP, or Microsoft DeepSpeed.

Highlights

• Performance Gains: Increases multi-GPU training throughput by up to 20% and reduces memory usage by up to 60%.
• Post-Training Optimization: Specialized kernels for alignment and distillation (DPO, ORPO, SimPO, etc.) that can reduce memory usage by up to 80%.
• Broad Compatibility: Works with Flash Attention, PyTorch FSDP, DeepSpeed, and various trainer frameworks like Axolotl and LLaMA-Factory.
• Hardware Support: Compatible with NVIDIA (CUDA), AMD (ROCm), and Ascend NPU.
• Exact Computation: Ensures accuracy through rigorous unit tests and convergence testing, providing exact results rather than approximations.