NVIDIA/NVSentinel

What it solves

NVSentinel addresses the challenge of GPU hardware failures in Kubernetes clusters, which can lead to silent data corruption, failed training jobs, or wasted compute resources. It automates the detection, isolation, and repair of faulty GPU nodes to maintain cluster reliability.

How it works

NVSentinel operates as a modular system within Kubernetes. It uses various health monitors (such as DCGM for GPU metrics, syslog, and cloud provider events) to detect faults. When a fault is identified, it coordinates through a central event store to cordon the node, drain active workloads to prevent job failure, and trigger automated remediation actions like GPU resets or full node reboots. It also includes a preflight check feature to prevent new jobs from being scheduled on unhealthy hardware.

Who it’s for

This tool is designed for cluster administrators and MLOps engineers managing large-scale NVIDIA GPU infrastructure on Kubernetes who need to automate node maintenance and improve uptime for AI training and inference workloads.

Highlights

• Real-time detection of GPU, NIC, and system-level faults.
• Automated protection by cordoning and draining nodes before failures impact jobs.
• Self-healing capabilities through automated GPU resets or node reboots.
• Extensible architecture allowing for custom health monitors and remediation strategies.
• Preflight checks to ensure hardware readiness before job execution.