Tencent/AngelSlim

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

What it solves

AngelSlim is a comprehensive toolkit designed to make large model compression more accessible and efficient. It addresses the challenge of deploying massive AI models (like LLMs, VLMs, and Diffusion models) on limited hardware by providing a unified framework for reducing their size and increasing their inference speed without significant performance loss.

How it works

The toolkit integrates multiple advanced compression strategies into a single, easy-to-use framework:

  • Quantization: Supports a wide range of formats including FP8, INT8, INT4 (GPTQ/AWQ), and ultra-low bit-widths like 1.25-bit (Sherry) and ternary quantization (Tequila).
  • Speculative Decoding: Uses draft models to predict tokens, which are then verified by the target model. It includes the AngelSpec framework for training these draft models using a disaggregated design where inference and training run on separate GPU pools.
  • Sparse Attention: Implements algorithms like Stem and MInference to accelerate the prefill stage of long-context models by dynamically selecting key blocks.
  • Distillation: Provides quantization-aware distillation to transfer knowledge from larger models to smaller, compressed versions.
  • Model-Specific Optimizations: Includes specialized techniques for different modalities, such as token pruning for VLMs and caching (DeepCache, TeaCache) for Diffusion models.

Who it’s for

It is intended for AI developers and researchers who need to optimize large-scale models for deployment on various hardware, from high-end servers to consumer-grade laptops (e.g., running 214GB models on a 4090 GPU).

Highlights

  • Broad Model Support: Works across LLMs (Hunyuan, Qwen, DeepSeek), VLMs (Qwen-VL), Diffusion models (FLUX, SDXL), and Speech models.
  • Crossover Compatibility: Integrates with popular backends like vLLM, SGLang, and HuggingFace.
  • Extreme Compression: Capable of compressing models from terabytes to hundreds of gigabytes with minimal performance drops.
  • Integrated Training: Includes AngelSpec, a torch-native framework for training speculative-decoding draft models with support for long-sequence training and multi-node scaling.

Related

  • Project
    vllm-project/llm-compressor

    LLM Compressor is a Python library that quantizes and prunes large language models into the `compressed‑tensors` format, enabling memory‑efficient deployment with vLLM. It supports many low‑precision formats (NVFP4, FP8, INT4, etc.), several PTQ/GPTQ algorithms, DDP and disk‑offloading for huge models, and ships pre‑quantized checkpoints for popular LLMs.

  • Project
    NVIDIA/Model-Optimizer

    A library for accelerating AI models through quantization, pruning, and distillation, enabling faster inference and reduced memory usage across various deployment frameworks.

  • Project
    FujitsuResearch/OneCompression

    A Python package for automated LLM compression that uses advanced quantization techniques like QEP and AutoBit to reduce model size and VRAM usage.

  • Project
    intel/auto-round

    AutoRound is Intel’s open‑source Python toolkit for ultra‑low‑bit (2‑4 bit) quantization of large language and vision‑language models. It offers high‑accuracy rounding (sign‑gradient descent), multiple export formats (AutoRound, AutoAWQ, AutoGPTQ, GGUF), and seamless integration with Transformers, vLLM, SGLang, and LLM‑Compressor. Install via `pip install auto-round` (or the HPU/XPU variants) and quantize models through a simple CLI (`auto-round …`) or Python API (`AutoRound(...).quantize_and_save`).

  • Project
    datawhalechina/diy-llm

    A Chinese‑language, open‑source curriculum that teaches you to build, train, and evaluate large language models from scratch, complete with lecture notes, six hands‑on assignments, and runnable PyTorch code.