alibaba/clusterdata

What it solves

This project provides researchers and students with real-world production data from Alibaba's large-scale data centers. It helps solve the challenge of understanding how modern internet data centers operate, specifically regarding how online services and batch workloads are managed and collocated on the same hardware.

How it works

The program releases anonymized datasets (traces) captured from Alibaba's production clusters. These traces include detailed metrics such as CPU performance, memory usage, microservice dependencies, GPU utilization, and job scheduling information. Researchers can download these datasets to simulate production environments, test new scheduling algorithms, or analyze workload characteristics to improve resource efficiency.

Who it’s for

It is designed for researchers, students, and practitioners in cloud computing, distributed systems, and data center management who need realistic, large-scale production data to validate their ideas, simulations, and academic papers.

Highlights

• Provides multiple versions of historical and recent cluster traces, including GPU-specific and microservice-focused datasets.
• Includes detailed documentation, schemas, and processing scripts for various data types.
• Supports research into complex topics like workload characterization, resource contention, and scheduling algorithms.
• Facilitates the study of modern infrastructure, including AI/ML workloads, serverless computing, and GenAI serving systems.