meta/FAI-PEP
Facebook AI Performance Evaluation Platform
What it solves
Machine learning is moving fast: new frameworks, new hardware, new software backends, and new models appear constantly. Comparing the runtime performance of a model or a backend quickly becomes a mess because every combination tests differently. This project solves that by giving you one place to measure and compare inference performance across many setups. The results guide decisions on where to invest in framework development, backend optimization, hardware selection, and model iteration.
How it works
FAI-PEP splits benchmarking into three connected parts. First, there are centralized model and benchmark specifications written in JSON. These specs include download links and MD5 hashes so models are fetched automatically and verified; if a model changes, only the spec needs updating. Second, a central benchmark driver runs the actual tests on each backend. The same driver code is used everywhere to reduce variation, and it can distribute tests to different machines because each backend has its own build and run environment. Third, the results go to a central consumption point where you can compare performance across all backends and models at once.
The platform collects several metrics: latency (for the whole network or individual operators), error (difference between outputs and golden reference outputs), energy/power (per inference, typically on a phone without battery), and any custom metrics your own binary produces. After a run, you can check for performance regressions between different code commits using A/B testing: you run the same tests on an older commit (control) and a newer commit (treatment) at the same time, then compare the relative performance difference. This removes most environmental variation and makes regression detection more precise.
To extend the platform, you add a new framework by creating a directory under benchmarking/frameworks, deriving from framework_base.py, and implementing its methods. Similarly, a new backend goes under benchmarking/platforms, deriving from platform_base.py. The run entry points are run_bench.py (single interactive run), harness.py (standalone benchmark run), and repo_driver.py (continuous runs, repeatedly pulling code, building, and benchmarking).
Who it’s for
This project targets developers of ML frameworks, software backends, and hardware accelerators, as well as machine learning engineers who need to pick the fastest model or backend for their use case. It is also useful for anyone responsible for monitoring performance across many model/backend combinations and catching regressions early. The supported frameworks and backends currently include Caffe2, TFLite, CPU, GPU, DSP, Android, iOS, and Linux-based systems, but the design is meant to be extended.
Highlights
- Framework- and backend-agnostic: compare performance across many setups from one platform.
- Centralized specifications for models and benchmarks, with automatic download and MD5 verification.
- Collects latency, error, energy/power, and custom metrics from your own binaries.
- A/B testing methodology for regression detection between commits, reducing environmental noise.
- Extensible design: add new frameworks and backends by deriving from base classes.
- Multiple entry points: single-run harness, interactive driver, and continuous benchmark loop.
- Distributed execution to handle unique build/run environments per backend.
Related
- Project
- Project
- Project
- Project