leptonai/gpud

GPUd automates monitoring, diagnostics, and issue identification for GPUs

What it solves

GPUd addresses the challenges of maintaining GPU efficiency and reliability in large-scale AI/ML clusters. It prevents production downtime by actively monitoring GPU health and managing workloads to ensure hardware is performing optimally.

How it works

GPUd runs as a lightweight, self-contained binary on Linux machines. It integrates with the NVIDIA ecosystem (including NVML and DCGM) and system tools like Docker, containerd, and Kubernetes to collect and report critical metrics. It monitors GPU power, temperature, and fabric status, while scanning for hardware slowdowns, kmsg errors, and NVML Xid events to detect failures before they impact workloads.

Who it’s for

It is designed for engineers and operators managing GPU clusters for AI/ML workloads, particularly those using NVIDIA hardware in production environments or integrating with DGX Cloud Lepton.

Highlights

  • GPU-Centric Monitoring: Provides a unified view of critical GPU and GPU fabric metrics.
  • Production Proven: Used in the production infrastructure of DGX Cloud Lepton.
  • Low Overhead: Designed to keep monitoring off the critical path of the workload to minimize CPU and memory usage.
  • Flexible Deployment: Supports direct host installation via systemd, foreground/background execution, and Kubernetes deployment via Helm charts.

Related

  • Project
    NVIDIA/DCGM

    A suite of tools for managing and monitoring NVIDIA data-center GPUs in cluster environments, providing health monitoring, diagnostics, and governance policies.

  • Project
    psalias2006/gpu-hot

    A lightweight, self-hosted real-time NVIDIA GPU monitoring dashboard that supports single-machine or multi-node cluster tracking.

  • Project
    wentbackward/nv-monitor

    A lightweight, zero-dependency monitoring tool for NVIDIA GPU systems that provides a TUI, CSV logging, and a Prometheus exporter for tracking CPU, GPU, and memory metrics.

  • Project
    anderskm/gputil

    A Python module that uses nvidia-smi to monitor NVIDIA GPU status and programmatically select available GPUs based on memory and load.

  • Project
    cfregly/ai-performance-engineering

    A comprehensive collection of code and resources for optimizing AI systems, focusing on GPU tuning, distributed training, and high-throughput inference scaling.