PaddleFormers – Transformers on PaddlePaddle

What it is – PaddleFormers is a library that brings the Hugging Face Transformers‑style API to Baidu’s deep‑learning framework PaddlePaddle. It focuses on training and fine‑tuning large language models (LLMs) and visual‑language models (VLMs) with high‑performance distributed strategies (tensor, pipeline, expert parallelism, FP8, etc.) and supports a wide catalog of state‑of‑the‑art models.

---

Key Features (as described in the README)

• Broad model coverage – 100+ LLM and VLM families (DeepSeek‑V3/V4, GLM‑4.5, Qwen 2/3/3.5, Llama 3/4, ERNIE‑4.5, Gemma‑3, etc.).
• Performance‑first training – FP8 low‑precision, FlashMLA attention, highly optimized communication/computation overlap, and expert‑parallel libraries (DeepEP/HybridEP). Claims to beat Megatron‑LM on several models.
• Full training pipeline – Pre‑training, instruction‑tuning (SFT/LoRA), preference alignment (DPO/LoRA), and post‑training capabilities (CPT, SFT‑LoRA, DPO‑LoRA, function‑call, reasoning, grounding, etc.).
• Safetensors support – Models are saved in the interoperable Safetensors format, usable with FastDeploy, vLLM, SGLang, etc.
• Hardware flexibility – Native support for Chinese domestic accelerators (Kunlun P800, TianShu TianGai 150, MuXi C550, MUSA, etc.) with dedicated installation guides.
• API compatibility – Same class names (AutoTokenizer, AutoModelForCausalLM, etc.) as Hugging Face, making migration straightforward.

---

Installation (quick summary)

1. Prerequisites – Python ≥ 3.12, CUDA ≥ 13.0, PaddleFleet ≥ 0.3 for GPU training.
2. Docker (recommended) – Pull the official PaddlePaddle GPU image and run a container with the repository mounted.
3. Pip / source –

   # From source (editable install with training deps)
   git clone https://github.com/PaddlePaddle/PaddleFormers.git && cd PaddleFormers
   python -m pip install -e '.[paddlefleet]' \
       --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu126/ \
       --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
   

   Or install the binary package for inference‑only use:

   python -m pip install paddleformers
   

4. Special hardware – Separate guides are provided for XPU, ILUVATAR‑GPU, Metax‑GPU, and MUSA.

---

Quick‑Start Code

from paddleformers.transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B-Base")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B-Base", dtype="bfloat16"
).eval()

inputs = tokenizer("请给我一段大模型的简短介绍：", return_tensors="pd")
output_ids = model.generate(**inputs, max_new_tokens=256)[0].tolist()[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))

Training can be launched with the CLI:

paddleformers-cli train ./examples/config/sft/full.yaml

---

Typical Use Cases

• Research & development – Pre‑train or continue‑train massive LLMs (e.g., DeepSeek‑V4 with 128K context) on multi‑GPU or multi‑node clusters.
• Instruction fine‑tuning – Apply SFT, LoRA, or DPO to adapt models for chat, reasoning, or function‑calling.
• Multimodal projects – Train VLMs such as Qwen‑VL or PaddleOCR‑VL using the same API.
• Deployment – Export to Safetensors and serve with FastDeploy, vLLM, or other inference engines.
• Domestic hardware – Leverage Kunlun, TianShu, or other Chinese GPUs for cost‑effective large‑model training.

---

Community & Governance

• Open‑source license – Apache‑2.0.
• Contribution – Guidelines in CONTRIBUTING.md; community encouraged to submit code and models.
• Support channels – Official WeChat QR code for a developer group, plus GitHub issues for troubleshooting.
• Acknowledgements – Built on design ideas from Hugging Face Transformers.

---

Bottom line: PaddleFormers is a full‑stack, high‑performance Transformers library for the PaddlePaddle ecosystem, targeting large‑scale LLM/VLM training, fine‑tuning, and deployment, with special attention to Chinese hardware and distributed training efficiency.