OpenAI Scaling Kubernetes to 7,500 Nodes

OpenAI has scaled a single Kubernetes cluster to 7,500 nodes to enable machine learning research teams to scale workloads without modifying their code. This infrastructure supports massive jobs where single pods often occupy entire nodes to maximize hardware efficiency through NVLink and GPUDirect.

Workload Characteristics

OpenAI's Kubernetes workloads differ significantly from typical enterprise applications. The primary focus is on large-scale machine learning jobs that prioritize hardware resource access over bin-packing or fragmentation.

  • Resource Utilization: Pods typically occupy an entire node to allow GPUs to communicate directly via NVLink and the NIC via GPUDirect, reducing the need for complex scheduling based on NUMA or CPU contention.
  • Job Nature: Jobs often run MPI (Message Passing Interface), making them "semi-stateful." If one pod in an MPI communicator dies, the entire job halts and restarts from the last checkpoint.
  • Networking Patterns: There is minimal reliance on Kubernetes load balancing, HTTPS traffic, or service discovery. Pods communicate directly via pod IP addresses using MPI over SSH.
  • Storage: The system relies primarily on blob storage for streaming datasets and checkpointing, using PersistentVolumes only when POSIX semantics are required.

Networking Optimizations

To support 7,500 nodes and approximately 200,000 concurrent IP addresses, OpenAI moved away from Flannel to native pod networking for Azure VMSS (Virtual Machine Scale Sets) and relevant CNI plugins.

High-Throughput Networking

Using alias-based IP addressing instead of route-based networking avoided limitations in the number of effective routes and provided host-level network throughput. This approach eliminates encapsulation, simplifying the setup and avoiding packet fragmentation issues related to MTU.

Network Monitoring

OpenAI uses iptables mangle rules to tag packets as either internal or internet-bound. These counters are tracked via the open-source iptables-exporter and integrated into Prometheus, allowing researchers to visualize network bottlenecks.

Cluster Isolation

Clusters are organized in a hub-and-spoke model. While researchers can access individual clusters (spokes) via the hub, clusters cannot communicate with each other, ensuring failure isolation.

API Server and etcd Stability

To maintain cluster health at scale, OpenAI runs API servers and etcd on dedicated nodes outside the cluster.

  • Deployment: The largest clusters utilize five API servers and five etcd nodes to distribute load.
  • Resource Usage: API server memory usage scales linearly with cluster size, reaching up to 70GB of heap for 7,500 nodes.
  • EndpointSlices: The adoption of EndpointSlices (introduced in Kubernetes 1.17) reduced the load from WATCHes on Endpoints by 1,000x, preventing $N^2$ bandwidth spikes when nodes were added or removed.
  • Scaling Strategy: To avoid overloading API servers, OpenAI avoids using DaemonSets for API interactions and smooths out the addition of new nodes during autoscaling events.

Monitoring with Prometheus and Grafana

OpenAI uses Prometheus for time-series metrics and Grafana for visualization, but encountered significant scaling hurdles.

  • Memory Leaks: A bug in the /api/v1/series API caused Prometheus to crash due to unbounded memory consumption when Grafana queried histogram metrics. OpenAI patched Prometheus to enforce a timeout using a Context.
  • WAL Replay Performance: Prometheus startup times were delayed by hours due to Write-Ahead-Log (WAL) replay. Setting GOMAXPROCS=24 mitigated this by reducing CPU contention on high-core servers.
  • Metric Filtering: To manage the volume of data, Prometheus rules are used to "drop" granular or unused metrics from ingestion.

Node Health and GPU Validation

Automation is used to detect and remove misbehaving nodes through a combination of passive and active healthchecks.

Passive Healthchecks

Systems monitor network reachability, disk health, and GPU errors (such as Uncorrectable ECC errors) via the dcgm-exporter and NVML Device Query API. Failing nodes are automatically cordoned; serious failures trigger pod eviction and eventual VM termination.

Active GPU Tests

Because some GPU issues do not trigger error codes, OpenAI uses a "preflight" system. New nodes join the cluster with a taint and label; a DaemonSet runs exhaustive GPU tests before removing the taint to allow general workloads.\n

Resource Management and Scheduling

OpenAI implemented custom mechanisms to handle resource allocation and gang scheduling for competing research teams.

  • Team Taints: A team-resource-manager service uses taints (openai.com/team=teamname:NoSchedule) and admission webhooks to allocate specific node capacities to research teams while allowing low-priority pods to borrow unused capacity.
  • Resource Balloons: To prevent the cluster-autoscaler from removing idle nodes (which would increase spin-up latency and API server strain), OpenAI uses "balloon" Deployments. These low-priority pods occupy space but are immediately evicted when actual work arrives.
  • Gang Scheduling: To prevent deadlocks where multiple experiments each take half the required cluster capacity, OpenAI uses the Coscheduling plugin (introduced in Kubernetes 1.18) to ensure all pods in a StatefulSet are scheduled before training begins.

Remaining Challenges

OpenAI identifies two primary unsolved problems:

  1. Metrics Storage: The Prometheus TSDB engine is slow to compact and prone to "too many samples" errors. OpenAI is migrating to a different Prometheus-compatible storage and query engine.
  2. Traffic Shaping: The aggregate internet bandwidth used by researchers can strain external datasets and software package repositories, necessitating better pod network traffic shaping.

Sources