nebius/soperator

What it solves

Soperator simplifies the deployment and management of Slurm clusters on Kubernetes, specifically for AI training and high-performance computing (HPC). It addresses the manual effort required to set up and maintain Slurm, the risk of long-running training jobs failing due to hardware issues, and the inefficiency of idle GPUs caused by fixed-size clusters or poor placement.

How it works

Soperator uses the Kubernetes operator pattern to manage Slurm. Users define a SlurmCluster custom resource that describes the entire cluster layout (controllers, workers, storage, etc.). The operator then reconciles this specification into Kubernetes objects like Deployments and StatefulSets. To ensure consistency across nodes, it uses a "jail" (a shared root filesystem via a PVC) so that configuration and package changes are applied cluster-wide immediately. It also implements passive and active health checks to automatically drain and replace failing nodes.

Who it’s for

It is designed for platform teams and engineers who need to provide Slurm environments for AI training without manual per-node management, as well as teams migrating from bare-metal Slurm installations to Kubernetes.

Highlights

• Declarative Management: Manage the entire Slurm stack, including NVIDIA drivers, CUDA, and NCCL, via a single Kubernetes resource.
• Automated Reliability: Active and passive health checks detect GPU and network failures, triggering automatic node replacement.
• GPU Efficiency: Supports ephemeral nodes for autoscaling and InfiniBand-aware placement to optimize GPU utilization.
• Portable Deployment: Compatible with various cloud providers (AWS, GCP, Azure, Nebius, OCI) and bare-metal or air-gapped environments.
• Standard Slurm Experience: Maintains familiar Slurm behaviors like gang scheduling, fair-share, and preemption.