NVIDIA/dcgm-exporter

What it solves

DCGM-Exporter allows users to monitor NVIDIA GPU performance and health by exposing GPU metrics in a format that Prometheus can collect and visualize (e.g., via Grafana). It bridges the gap between NVIDIA's Data Center GPU Manager (DCGM) and cloud-native monitoring stacks.

How it works

The tool leverages NVIDIA DCGM to gather telemetry data from GPUs. It runs as a container or a systemd service, exposing a /metrics endpoint that Prometheus scrapes. It can be deployed on bare metal or within Kubernetes clusters (often as part of the NVIDIA GPU Operator).

Who it’s for

It is designed for system administrators, DevOps engineers, and ML platform engineers who manage GPU-accelerated workloads in data centers or Kubernetes environments.

Highlights

• Prometheus Integration: Native support for exporting GPU metrics to Prometheus.
• Flexible Configuration: Users can define which metrics to collect via custom CSV or YAML configuration files.
• Kubernetes Ready: Easy deployment via Helm charts and integration with the NVIDIA GPU Operator.
• Advanced Telemetry: Supports tracking SM clock frequencies, memory temperatures, and cumulative XID errors.
• Contextual Labeling: Ability to add container runtime labels and HPC job mapping to associate GPU usage with specific jobs or containers.