facebookresearch/HolisticTraceAnalysis

Holistic Trace Analysis (HTA)

What it is – HTA is a Python library that helps researchers and engineers understand why a distributed PyTorch training job is slow. It reads the trace files produced by the PyTorch Profiler (Kineto) and turns them into a set of ready‑to‑use data‑frames and visualisations that pinpoint where GPUs spend time.

Key capabilities

Feature	What you get
Temporal breakdown	Per‑GPU time split into compute, communication, memory ops and idle periods.
Kernel breakdown & distribution	Lists the longest‑running CUDA kernels per rank and shows how their durations vary across ranks.
Idle‑time analysis	Shows why a GPU is idle (waiting for the host, waiting for another kernel, or unknown).
Comm‑compute overlap	Percentage of time communication overlaps useful computation.
Frequent kernel patterns	Finds the most‑repeated CUDA kernels for a given PyTorch operator (e.g., aten::linear).
Kernel launch statistics	Histograms of very short, very long, and unusually delayed kernel launches.
Augmented counters	Adds queue‑length and memory‑bandwidth traces to the original profile for roof‑line style analysis.
Trace comparison	Side‑by‑side diff of two runs to see what changed.
CUPTI counter API (experimental)	Pulls low‑level GPU performance counters for deeper roof‑line analysis.

Who it’s for – Anyone training large models on multiple GPUs/hosts who wants a systematic, scriptable way to locate bottlenecks without manually digging through raw profiler output.

Installation

# From PyPI (recommended)
pip install HolisticTraceAnalysis

# Or from source
git clone https://github.com/facebookresearch/HolisticTraceAnalysis.git
cd HolisticTraceAnalysis
git submodule update --init
pip install -r requirements.txt
pip install -e .

Runs on Linux and macOS with Python ≥ 3.10.

Typical workflow

1. Collect traces with the PyTorch profiler (torch.profiler.profile(..., record_shapes=True, with_stack=True, ...)).
2. Put all trace files in one folder – HTA expects a single directory per run.
3. Open a notebook (or a Python script) and create a TraceAnalysis object:

   from hta.trace_analysis import TraceAnalysis
   analyzer = TraceAnalysis(trace_dir="/path/to/traces")
   

4. Call the methods you need, e.g.:

   temporal = analyzer.get_temporal_breakdown()
   kernels  = analyzer.get_gpu_kernel_breakdown()
   idle     = analyzer.get_idle_time_breakdown()
   overlap  = analyzer.get_comm_comp_overlap()
   

   Each returns a pandas DataFrame that can be plotted or inspected.
5. For deeper dives, use the demo notebooks in examples/ (trace_analysis_demo.ipynb, trace_diff_demo.ipynb).

Configuration – Logging is controlled via hta/configs/logging.config; you can change the level or point to a custom config file.

Documentation & support – Full API docs are hosted at https://hta.readthedocs.io. The project is open‑source under the MIT license and welcomes contributions via pull requests.

---

Bottom line – HTA turns raw PyTorch profiler traces into actionable performance reports, making it easier to optimise distributed training pipelines.