facebookincubator/dynolog

What it solves

Dynolog solves the problem of monitoring and profiling large-scale AI training workloads that run across many machines and use both CPUs and GPUs. When training big models, it's hard to see where bottlenecks are — whether the GPU is idle waiting on the CPU, the network, or the disk. Existing tools monitor CPUs and GPUs separately, so you can't get a holistic view. Dynolog is a lightweight daemon that runs continuously on each machine, collects performance metrics from the CPU, GPU, and Linux kernel, and can also trigger deep-dive profiling on demand — for example, capturing PyTorch traces across hundreds of GPUs at once.

How it works

Dynolog runs as a background service (daemon) on Linux systems. It has two main parts: the dynolog server that collects data, and the dyno command-line tool that talks to the server locally or remotely.

The server supports two modes:

• Always-on monitoring: continuously collects system metrics (CPU, network, I/O), CPU hardware performance events (like instructions and cycles per second), and NVIDIA GPU metrics via NVIDIA's DCGM library.
• Deep-dive profiling: activated on demand through a remote procedure call. For PyTorch, you enable a flag (--enable_ipc_monitor) and set an environment variable (KINETO_USE_DAEMON=1), then use dyno gputrace to capture traces from specific processes — even across remote nodes or a distributed training job.

Metrics are logged to files or exported to observability backends (like Grafana, Datadog, New Relic) using the OpenTelemetry Protocol (OTLP). The daemon is designed to be lightweight so it doesn't slow down production workloads.

Who it's for

Dynolog is aimed at engineers and researchers who train or run large AI models on GPU clusters — especially those using PyTorch — and need to debug performance issues in production or distributed environments. It's also useful for anyone running heterogeneous CPU-GPU systems on Linux who wants a unified monitoring and profiling tool.

Highlights

• On-demand PyTorch profiling: trigger traces remotely with a single command, even across hundreds of GPUs or distributed training jobs.
• Unified CPU+GPU monitoring: collects Linux kernel metrics, CPU hardware performance events (Intel/AMD), and NVIDIA GPU metrics (via DCGM) in one place.
• OpenTelemetry export: sends metrics to any OTLP-compatible backend like Grafana, Datadog, or New Relic.
• Lightweight and production-friendly: designed to run continuously without causing performance degradation.
• Easy installation: available as RPM/Debian packages, with a userspace mode for systems without root access.