PaddlePaddle/PaddleFormers

PaddleFormers is an easy-to-use library of pre-trained large language model zoo based on PaddlePaddle.

What it solves

PaddleFormers provides a high-performance training library for Large Language Models (LLMs) and Vision Language Models (VLMs) within the PaddlePaddle ecosystem. It aims to offer a model interface and functional experience equivalent to Hugging Face Transformers while optimizing for high-performance distributed training, reducing resource consumption, and simplifying the complex underlying optimization details for the user.

How it works

Built on the PaddlePaddle deep learning framework, the library implements a wide array of distributed training strategies, including tensor parallelism, pipeline parallelism, and expert parallelism. It utilizes advanced acceleration techniques such as automatic mixed precision, FP8 low-precision training, and communication-computation overlap. For specific architectures like DeepSeek-V4, it incorporates specialized operators like FlashMLA and high-performance FP8 MoE modules to maximize efficiency.

Who it’s for

AI researchers and developers who need to train or fine-tune large-scale models (LLMs and VLMs) using the PaddlePaddle framework, particularly those utilizing domestic Chinese hardware (such as Kunlunxin, Iluvatar, Metax, and Moore Threads) or seeking performance that exceeds Megatron-LM in specific scenarios.

Highlights

  • Extensive Model Support: Supports over 100 mainstream models, including DeepSeek-V3/V4, GLM-4.5, Qwen series, and ERNIE series.
  • Full-Lifecycle Training: Supports the entire pipeline from pre-training and continued pre-training (CPT) to post-training techniques like SFT, SFT-LoRA, DPO, and DPO-LoRA.
  • Hugging Face Compatibility: API design is highly consistent with Hugging Face Transformers, and it supports the Safetensors format for seamless weight transfer to tools like vLLM or SGLang.
  • High-Performance Optimizations: Implements Data Packing and Padding Free techniques to optimize data flow and training speed.
  • Broad Hardware Adaptation: Deeply adapted for various domestic Chinese AI chips beyond standard NVIDIA GPUs.

Related

  • Project
    baidu-baige/LoongForge

    A high-performance training framework for LLMs, VLMs, diffusion, and embodied models that optimizes parallelism and kernel efficiency to reduce training costs.

  • Project
    PaddlePaddle/PaddleNLP

    A comprehensive LLM development suite based on PaddlePaddle that provides tools for efficient training, lossless compression, and high-performance inference across multiple hardware platforms.

  • Project
    PaddlePaddle/FastDeploy

    A production-grade deployment toolkit for LLMs and VLMs based on PaddlePaddle, offering high-performance inference and broad hardware compatibility.

  • Dispatch
    PaddlePaddle Integration with Hugging Face Hub

    Hugging Face has partnered with PaddlePaddle to integrate its deep learning platform and libraries, starting with PaddleNLP, into the Hugging Face Hub for improved accessibility and sharing.

  • Project
    vllm-project/llm-compressor

    LLM Compressor is a Python library that quantizes and prunes large language models into the `compressed‑tensors` format, enabling memory‑efficient deployment with vLLM. It supports many low‑precision formats (NVFP4, FP8, INT4, etc.), several PTQ/GPTQ algorithms, DDP and disk‑offloading for huge models, and ships pre‑quantized checkpoints for popular LLMs.