flagos-ai/FlagPerf

What it solves

FlagPerf is an integrated AI hardware evaluation engine designed to measure the actual capabilities of AI hardware within a complete software stack (model + framework + compiler). It moves beyond simple "time-to-complete" metrics to provide a comprehensive, industry-oriented assessment of how hardware performs in real-world AI training and inference scenarios.

How it works

FlagPerf evaluates hardware across multiple dimensions and environments:

• Multi-dimensional Metrics: It measures functional correctness (whether a chip can support a specific model), performance, resource utilization, and ecosystem adaptation.
• Diverse Workloads: It includes over 30 classic models and 80 training examples across computer vision (CV), natural language processing (NLP), speech, and multimodal domains, including large language model (LLM) training and inference.
• Software Integration: It connects hardware to the software ecosystem by supporting frameworks like PyTorch, TensorFlow, PaddlePaddle, and MindSpore, as well as inference engines like TensorRT, XTCL, IxRT, and TorchInductor.
• Scalable Testing: It tests performance across single-card, single-machine (typically 8 cards), and multi-machine environments.
• Fairness Controls: To ensure objective results, chip vendors are only permitted to modify hardware-related code (e.g., distributed communication, batch size) for adaptation, while the final testing is conducted by the BAAI (Beijing Academy of Artificial Intelligence) using the open-source FlagPerf platform.

Who it’s for

• AI Hardware Vendors: To benchmark their chips against industry standards and verify software stack compatibility.
• AI Infrastructure Engineers: To evaluate the performance and stability of different hardware/software combinations for training and deploying large models.
• Researchers: To obtain reproducible and fair performance data for AI accelerators.

Highlights

• Comprehensive Coverage: Supports a wide array of models including Llama 2/3, GPT-3, and Mixtral.
• Cross-Framework Support: Integrates multiple domestic and international training frameworks and inference engines.
• Multi-Scale Evaluation: Validates performance from a single chip up to multi-node clusters.
• Transparent Process: All test codes are open-source, ensuring the testing process and data are fully reproducible.