giovtorres/slurm-docker-cluster

What it solves

Slurm Docker Cluster provides a way to rapidly deploy a fully functional Slurm cluster using Docker Compose. It eliminates the complex manual setup of a high-performance computing (HPC) environment, allowing developers to test job scheduling, develop HPC applications, and manage compute resources without needing physical hardware clusters.

How it works

The project uses a multi-container architecture where each component of a Slurm environment is isolated in its own container. It includes a database (MySQL) for accounting, a controller (slurmctld) for scheduling, and scalable compute nodes (c1, c2, etc.) that use dynamic registration to join the cluster. It also provides optional components like a REST API for programmatic access, Open OnDemand for a web-based portal, and an Elasticsearch/Kibana stack for job monitoring.

Who it’s for

It is designed for developers and researchers who need a Slurm-compatible environment for development, testing, or lightweight usage, particularly those working with HPC software or AI/ML workloads that require job scheduling across multiple nodes.

Highlights

• Scalable Compute Nodes: Dynamically scale CPU and GPU workers without rebuilding the cluster.
• GPU Support: Integration with NVIDIA CUDA base images and the NVIDIA Container Toolkit for hardware acceleration.
• Integrated Software Management: Includes Spack and Lmod for easy installation and management of scientific software packages.
• Web-Based Management: Optional Open OnDemand portal for browser-based job submission and file management.
• Multi-Arch Support: Pre-built images available for both AMD64 and ARM64 architectures.