OpenAI Habitat Storage Platform Scaling: Multi‑Tenancy, Azure Cosmos DB Enhancements, and Global Performance

TL;DR

OpenAI unveiled the second part of its Habitat scaling story, describing how the team re‑architected Azure Cosmos DB usage, introduced a multi‑tenant default container, added hot‑key throttling, QoS, a custom serverless mode, Rust‑based services, and a CDC‑driven cache to serve over 500 PB of data at >20 M QPS with read latency cut from ~120 ms to ~10 ms.

Habitat Overview

Habitat is OpenAI’s online storage platform that provides millisecond‑scale data access for all OpenAI products across 40 geographic regions. It underpins more than 1 billion ChatGPT users and stores over 500 petabytes of data.

Multi‑Tenancy Design Journey

Conclusion: A shared‑container, multi‑tenant model now delivers reliable, low‑maintenance storage at scale.

OpenAI moved through three phases:

Phase I – Primitive Multi‑Tenancy – A single Cosmos DB container handled all traffic. Logical‑partition‑key limits and noisy‑neighbor effects caused throughput contention.
Phase II – Per‑Feature Single Tenancy – Separate containers isolated workloads but introduced massive capacity‑planning overhead and frequent launch‑day scaling incidents.
Phase III – Default Multi‑Tenancy with Mitigations – By collaborating with Azure, OpenAI added hot‑key throttling, QoS tiering, and a custom serverless mode, allowing a single shared container to safely host many tenants again.

"We have not had any provisioning or noisy‑neighbor incidents in the months since, despite accelerating product development." – OpenAI engineering team

Azure Cosmos DB Enhancements

Conclusion: Azure‑built primitives now handle hot keys, weak partitions, and scaling without manual intervention.

Hot‑Key Throttling – Prioritises throttling of the most active logical keys, preventing a single tenant from monopolising a physical partition.
Quality‑of‑Service (QoS) Tiering – Assigns lower priority to bulk or offline workloads, protecting interactive latency‑sensitive requests.
Custom Serverless Mode – Extends Azure’s serverless offering to 10 K RU/s per physical partition and adds RU‑based partition splitting, eliminating the need for manual throughput provisioning.

These changes let OpenAI run a single shared container that serves >20 M queries‑per‑second (QPS) while storing tens of petabytes.

Reliability Through Co‑Managed Services

Conclusion: Joint engineering with Azure provides production‑scale testing and rapid incident response.

OpenAI and Azure now share dashboards, test suites, and direct engineering channels.
A 2024 regional outage revealed that fail‑over of large Cosmos accounts could take hours; joint testing subsequently reduced fail‑over time and eliminated gateway stampedes.
Continuous co‑engineering of control‑plane scalability, connection pooling, and metadata caching has become standard practice.

Global Performance Optimizations

Conclusion: Read latency dropped 12× (≈120 ms → ≈10 ms) by moving data closer to users, replicating critical datasets, and adding a CDC‑driven cache.

Data Locality – Multiple Cosmos DB accounts are deployed in regions with data‑residency requirements; where capacity permits, data is colocated with compute.
Selective Replication – Latency‑critical datasets (e.g., auth metadata) are replicated to all 40 Azure regions; other data remains in fewer regions.
Integrated Distributed Cache – A Valkey‑based look‑aside cache stores recent objects near request handlers. Cache invalidation uses Cosmos DB change‑feed CDC, achieving seconds‑level freshness.
Rust Service Layer – Re‑writing the Habitat service in Rust removed Python overhead, contributing to the latency reduction.

Figure 5 (in the original post) shows the p50 read latency improvement from 120 ms to 10 ms after these changes.

Scaling Change Data Capture (CDC)

Conclusion: A dedicated CDC platform abstracts complexity and sustains millions of changes per second across thousands of containers.

CDC streams feed >100 Kafka topics, supporting secondary indexes, analytics, and real‑time features.
An internal control plane automatically discovers relevant Cosmos accounts/containers, balances worker fleets, and routes reads to replica regions.
Robustness measures include lease‑store isolation, dead‑letter queues, and custom Azure CDC SDK checkpoint batching.

Future Direction: Exabyte‑Scale In‑House Storage

Conclusion: OpenAI is prototyping a custom storage system to handle the next billion users.

The system combines RocksDB Cloud (from Rockset) with Raft‑based replication.
It retains the Cosmos‑style document model while using disaggregated storage to lower flash capacity needs.
Expected trade‑off: reduced peak QPS in exchange for massive storage efficiency.

Implications for the Industry

Multi‑Tenancy Viability – OpenAI demonstrates that, with provider‑level enhancements, a single shared container can safely serve billions of requests, reducing operational complexity.
Provider Collaboration – Deep co‑engineering between a consumer and cloud provider becomes essential at frontier scale; other large AI firms may need similar partnerships.
Performance‑First Architecture – Combining locality, selective replication, CDC‑driven caching, and low‑level language rewrites yields order‑of‑magnitude latency gains, a blueprint for any globally distributed service.
Roadmap Toward Self‑Built Storage – Even with Azure’s capabilities, OpenAI is investing in an in‑house solution, signaling that future frontier AI workloads may outgrow public cloud storage limits.

Key Takeaways

Habitat evolved from a simple Python client to a globally distributed, multi‑tenant storage platform serving >500 PB and >20 M QPS.
Azure Cosmos DB was extended with hot‑key throttling, QoS, and a custom serverless mode, enabling a single shared container to replace costly per‑feature containers.
Global latency improvements stem from data locality, selective replication, a CDC‑driven Valkey cache, and a Rust service rewrite, cutting read p50 latency from ~120 ms to ~10 ms.
Ongoing co‑engineering with Azure and a forthcoming exabyte‑scale in‑house storage system position OpenAI to support the next billion users.

For engineers interested in frontier‑scale data infrastructure, OpenAI is hiring for Habitat‑related roles.