nebius/soperator

What it solves

Soperator simplifies the deployment and management of Slurm clusters on Kubernetes, specifically for AI training and high-performance computing (HPC) workloads. It eliminates the manual effort required to set up, resize, and maintain Slurm clusters, while reducing the risk of long-running training jobs failing due to undetected hardware issues or inefficient GPU utilization.

How it works

Soperator uses the Kubernetes operator pattern. Users define a SlurmCluster custom resource that describes the cluster layout (controllers, login nodes, workers, storage, and health checks). The operator then reconciles this specification into Kubernetes objects like Deployments and StatefulSets. To ensure consistency across the cluster, it uses a "jail" (a shared root filesystem via a PVC) so that configuration and package changes are immediately visible to all nodes.

Who it’s for

It is designed for platform engineers and teams moving from bare-metal Slurm installations to Kubernetes-based environments who need a scalable, reliable way to provide Slurm scheduling for AI training.

Highlights

• Automated Cluster Management: Declarative updates for resizing, upgrades, and configuration via a single resource.
• High Reliability: Combines passive monitoring and active GPU/network probes to automatically drain and replace failed nodes.
• GPU Efficiency: Supports ephemeral nodes, autoscaling, and InfiniBand topology awareness to optimize GPU placement.
• Pre-installed Stack: Ships with NVIDIA drivers, CUDA, NCCL, and common training dependencies.
• Enterprise Integration: Supports SSSD for identity management, Prometheus/Grafana for observability, and OCI-compatible runtimes like Pyxis/Enroot.