Scaling Kubernetes to 2,500 Nodes

OpenAI has scaled its Kubernetes infrastructure to over 2,500 nodes on Azure using a combination of D15v2 and NC24 VMs to support deep learning research. This scaling effort required overcoming significant bottlenecks in state storage, pod scheduling, container image distribution, and network configuration.

Optimizing etcd Performance and Stability

Kubernetes state storage experienced critical latency and capacity issues as the cluster grew beyond 500 and 1,000 nodes.

Reducing Write Latency

When the cluster exceeded 500 nodes, kubectl timeouts occurred due to high write latency on network-attached SSDs. OpenAI resolved this by moving the etcd directory to local temp disks (SSDs connected directly to the instance), which reduced write latency from 2ms to 200us.

Managing API Server Load

At 1,000 nodes, high commit latency returned because kube-apiservers were reading over 500MB/s from etcd. This was caused by aggressive polling from monitoring processes (Fluentd and Datadog) querying the API servers from every node. Stability was restored by reducing the polling frequency of these processes.

Isolating Events and Storage Quotas

To prevent spikes in event creation from impacting the main cluster state, OpenAI stored Kubernetes Events in a separate etcd cluster using the --etcd-servers-overrides flag. Additionally, they increased the default 2GB etcd storage limit using the --quota-backend-bytes flag to prevent cascading failures where the autoscaler would terminate workers after etcd stopped accepting writes.

Kube Master and Scheduling Adjustments

OpenAI uses Kubernetes as a batch scheduling system and employs a custom autoscaler to minimize costs for idle nodes.

Bin-Packing Scheduling Policy

To ensure unused nodes can be terminated and large pods can be scheduled quickly, OpenAI replaced the default load-spreading policy with a "MostRequestedPriority" policy. This encourages bin-packing pods onto fewer nodes rather than spreading them evenly.

KubeDNS Reliability

The bin-packing policy created hotspots where some machines ran 10+ copies of KubeDNS, exceeding Azure VM external domain lookup limits (~200 QPS). OpenAI resolved this by implementing a pod anti-affinity rule to ensure KubeDNS pods are distributed across different hostnames.

Accelerating Docker Image Pulls

Large container images, such as a 17GB image used for the Dota project, caused pods to remain in a Pending state for extended periods.

Parallelizing Pulls and Timeout Tuning

By default, kubelet serializes image pulls (--serialize-image-pulls=true), meaning one large image could block all others. OpenAI set this to false and switched Docker to the overlay2 storage driver. They also moved the Docker root to instance-attached SSDs.

To prevent rpc error: code = 2 desc = net/http: request canceled errors caused by slow extraction of large images, they increased the --image-pull-progress-deadline to 30 minutes and set max-concurrent-downloads to 10 in the Docker daemon.

Preloading Common Images

To avoid hitting per-IP quota limits on the Google Container Registry (gcr.io) when nodes access it via NAT, OpenAI preloads the pod-infra-container-image and other common internal images directly into the worker machine images using docker image save and docker image load.

Networking and Kernel Tuning

High-scale distributed experiments revealed significant throughput drops and connectivity issues related to the network stack.

Overcoming Flannel Overhead

OpenAI observed that pods using Flannel were limited to ~2Gbit/s throughput, compared to 10-15Gbit/s between bare machines. To bypass this overhead for high-performance workloads, users can set hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet for their pods.

Resolving ARP Cache Overflows

Intermittent DNS resolution failures and connection hangs were caused by the kernel's ARP cache running out of space, as every pod in a large cluster consumes an ARP entry. This was identified by the neighbor table overflow! message in dmesg. OpenAI resolved this by increasing the ARP cache thresholds in /etc/sysctl.conf:

net.ipv4.neigh.default.gc_thresh1 = 80000
net.ipv4.neigh.default.gc_thresh2 = 90000
net.ipv4.neigh.default.gc_thresh3 = 100000