huggingface/gpu-fryer

What it solves

GPU fryer is designed to stress test NVIDIA GPUs to detect abnormal thermal throttling or performance degradation. This is particularly critical for ML inference and training workloads where the overall system performance is often limited by the slowest GPU in a cluster.

How it works

The tool creates two 8192x8192 matrices and performs continuous matrix multiplication using CUBLAS. To maximize stress, it allocates 95% of the GPU memory to write results in a ring buffer fashion. It uses NVML to monitor temperature and throttling status. Depending on the GPU's capabilities, it can use FP32, BF16, or FP8 precision to stress the Tensor Cores.

Who it’s for

It is intended for engineers and operators of HPC clusters and ML infrastructure, such as those used at Hugging Face, to ensure that all GPUs in a system are running at peak performance.

Highlights

• Multi-GPU Performance Checking: Checks for homogeneous performance across all GPUs in the system.
• Precision Options: Supports FP32, BF16, and FP8 precision modes.
• Thermal Monitoring: Reports temperature and hardware/software throttling status.
• Docker Support: Provides a ready-to-use Docker image for quick deployment.