Tencent/AngelSlim

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

What it solves

AngelSlim is a comprehensive toolkit designed to make large model compression more accessible, efficient, and integrated. It addresses the high computational and memory costs associated with deploying large-scale models (LLMs, VLMs, Diffusion, and Speech models) by providing a unified framework for reducing their size and increasing their inference speed without significant loss in quality.

How it works

The toolkit integrates various mainstream compression strategies into a single engine, allowing developers to apply different techniques based on the model type and desired outcome:

  • Quantization: Supports a wide range of formats including FP8 (Static/Dynamic), INT8, INT4 (GPTQ/AWQ/GPTAQ), NVFP4, and extreme low-bit quantization like Tequila (ternary) and Sherry (1.25-bit).
  • Speculative Decoding: Implements frameworks like Eagle3, SpecExit, and DFlare to accelerate token generation.
  • Sparse Attention & Pruning: Uses algorithms like Stem for block-sparse attention in long-context LLMs and IDPruner for vision token pruning in VLMs.
  • Distillation: Provides support for quantized distillation for both full-precision and QAT-style models.
  • Diffusion-Specific Optimizations: Includes FP8 quantization and caching mechanisms (DeepCache, TeaCache, TaylorCache) for image and video generation models.

Who it’s for

It is intended for AI developers and researchers who need to deploy large models on resource-constrained hardware (such as single GPUs or on-device environments) and those looking to optimize the end-to-end latency of model inference.

Highlights

  • Broad Model Support: Works across LLMs (Hunyuan, Qwen, DeepSeek, GLM), VLMs (Qwen-VL), Diffusion models (FLUX, SDXL, Wan), and Speech models (Qwen-Omni).
  • Unified Interface: Offers a "one-click" experience via config files or a Python API (Engine) to run complex compression workflows.
  • Extreme Quantization: Supports cutting-edge low-bit formats, including 1.25-bit and ternary quantization.
  • Deployment Ready: Provides scripts for deploying compressed models via industry-standard frameworks like vLLM and SGLang.