leptonai/gpud

GPUd automates monitoring, diagnostics, and issue identification for GPUs

What it solves

GPUd ensures GPU efficiency and reliability by actively monitoring hardware and managing AI/ML workloads. It addresses the need for a unified view of GPU health and performance metrics to prevent failures and optimize resource utilization in large-scale clusters.

How it works

GPUd is a self-contained binary that runs on Linux machines with a low footprint, keeping monitoring tasks off the workload critical path to minimize CPU and memory overhead. It integrates with Docker, containerd, Kubernetes, and the NVIDIA ecosystem (using tools like nvidia-smi, NVML Xid events, and DCGM) to collect and report GPU and GPU fabric metrics such as power, temperature, and hardware slowdowns.

Who it’s for

It is designed for operators of large-scale GPU clusters and AI infrastructure providers who need production-grade monitoring and reliability tools.

Highlights

  • GPU-Centric Monitoring: Provides a unified view of critical GPU and GPU fabric metrics.
  • Production Ready: Used in DGX Cloud Lepton's production infrastructure.
  • Easy Deployment: Supports installation via a simple script, systemd, or Helm charts for Kubernetes.
  • Comprehensive Error Detection: Detects critical hardware errors through kmsg, NVML Xid events, and DCGM.

Related

  • Project
    NVIDIA/DCGM

    A suite of tools for managing and monitoring NVIDIA data-center GPUs in cluster environments, providing health monitoring, diagnostics, and governance policies.

  • Project
    psalias2006/gpu-hot

    A lightweight, self-hosted real-time NVIDIA GPU monitoring dashboard that supports single-machine or multi-node cluster tracking.

  • Project
    wentbackward/nv-monitor

    A lightweight, zero-dependency monitoring tool for NVIDIA GPU systems that provides a TUI, CSV logging, and a Prometheus exporter for tracking CPU, GPU, and memory metrics.

  • Project
    Syllo/nvtop

    A htop-like task monitor for GPUs and AI accelerators that provides real-time visual monitoring of utilization, temperature, and power across multiple hardware vendors.

  • Project
    MiaAI-Lab/sparkDash

    sparkDash is a Docker‑based, real‑time web dashboard (React + Express) for monitoring NVIDIA DGX Spark clusters and any GPU‑enabled Linux/Windows host. It streams hardware stats (GPU/CPU, memory, network, energy) and LLM inference metrics (token‑per‑second, cache, latency) via SSH collectors, supports built‑in benchmarks, optional ComfyUI and Hermes Agent integration, Prometheus export, and a mobile‑friendly PWA.