NVIDIA/NVSentinel

What it solves

NVSentinel addresses the challenge of maintaining GPU infrastructure reliability in Kubernetes clusters. It automates the detection and remediation of hardware and software faults in GPU nodes, preventing faulty hardware from impacting AI workloads by automatically isolating and repairing them.

How it works

NVSentinel uses a microservices architecture that separates detection from action.

1. Detection: Pluggable health monitors (GPU, Syslog, Cloud Provider, and Kubernetes Object monitors) detect faults and send events via gRPC to Platform Connectors.
2. Persistence: Platform Connectors validate these events and store them in a MongoDB database, while updating Kubernetes node conditions.
3. Remediation: Independent core modules watch the MongoDB change streams. Depending on the fault, they trigger specific actions: the Fault Quarantine module cordons the node, the Node Drainer evicts workloads, and the Fault Remediation module creates maintenance CRDs to trigger external break-fix workflows.

Who it’s for

It is designed for platform engineers and cluster administrators managing large-scale NVIDIA GPU clusters on Kubernetes who need to automate GPU health management and reduce manual intervention during hardware failures.

Highlights

• Automated Lifecycle: Handles the full cycle from fault detection to node cordoning, workload draining, and triggering repairs.
• Lighweight Integration: Uses gRPC interfaces for health monitors and MongoDB change streams for asynchronous coordination between modules.
• Broad GPU Support: Validated across multiple NVIDIA architectures including Hopper, Ampere, Blackwell, and Ada Lovelace.
• Extensible Design: Supports custom health monitors and drain plugins via a modular architecture.