microsoft/TransformerCompression

What it solves

It reduces the size and memory footprint of transformer networks and Large Language Models (LLMs) without requiring complex code optimizations. This allows models to run faster and use less memory while maintaining as much performance as possible.

How it works

SliceGPT uses a post-training sparsification scheme. It first applies orthogonal transformations to each transformer layer that leave the model's behavior unchanged. It then identifies and "slices off" the least-significant rows and columns of the weight matrices based on eigenvalue decay. This replaces large weight matrices with smaller, dense ones, effectively reducing the model's embedding dimension.

Who it’s for

Developers and researchers looking to compress LLMs for more efficient deployment, particularly those using models from the Hugging Face hub like Llama-2, Llama-3, Phi-2, and OPT.

Highlights

• Post-training compression: Works on existing models without needing to retrain from scratch.
• Reduced memory and speedups: Decreases the embedding dimension to achieve faster inference and lower memory usage.
• Recovery fine-tuning: Supports post-slicing recovery fine-tuning (RFT) using LoRA to regain lost performance.
• Extensible architecture: Provides a framework (ModelAdapter and LayerAdapter) to add support for new Hugging Face model types.