facebook/FAI-PEP

What it solves

FAI-PEP provides a standardized way to benchmark machine learning inference performance across different hardware backends and software frameworks. It eliminates the difficulty of manually evaluating how new model iterations, framework updates, or hardware choices affect runtime metrics, ensuring that performance comparisons are fair and consistent.

How it works

The platform uses a centralized system to manage model and benchmark specifications in JSON format, ensuring that the same inputs are used for all comparisons. It employs a distributed execution model where a centralized driver manages the running of benchmarks across various environments (such as Android, iOS, Linux, CPU, GPU, and DSP).

To detect performance drops, it uses an A/B testing methodology that compares a new commit (treatment) against an older commit (control) to isolate the relative performance difference and remove environmental variations.

Who it’s for

• Framework Developers: To optimize software backends and evaluate framework enhancements.
• Hardware Engineers: To select and test the most efficient hardware solutions for ML models.
• ML Researchers: To iterate on models and verify their runtime performance.

Highlights

• Backend Agnostic: Supports a wide range of platforms including CPU, GPU, DSP, Android, iOS, and Linux.
• Framework Support: Compatible with Caffe2 and TFLite.
• Comprehensive Metrics: Tracks latency (delay), output error, and energy/power consumption.
• Regression Detection: Automates the detection of performance regressions between commits using A/B testing.
• Extensible Architecture: New frameworks and platforms can be added by inheriting from base classes (framework_base.py and platform_base.py).