OpenAI Habitat scaling to serve over 1 billion ChatGPT users

TL;DR

OpenAI unveiled that its Habitat storage platform now processes >70 M requests / s, stores >500 PB, and supports >1 B weekly ChatGPT users, after evolving from a simple Python client library to a distributed service now being rewritten in Rust for efficiency.

Overview of Habitat’s Growth

Habitat now handles more than 70 M requests per second, serves over 1 B weekly users across ~40 regions, and stores >500 PB of data. The system began in mid‑2024 as a thin Python client library over Azure Cosmos DB and has become a complex, globally distributed storage service.

Why a Service Replaced the Library

Coordinating protocol changes across dozens of services became brittle, prompting a move to a centralized service. The client‑side library required feature‑flag rollouts, shadowing, and multi‑region routing logic that took days to coordinate and still caused outages. Centralizing storage logic gave OpenAI a single point for deployments, observability, security, and policy enforcement.

Architectural Choices and Trade‑offs

Python Service as a Strategic Incursion

  • Running Habitat as a Python service introduced higher latency and CPU/memory costs, but allowed rapid delivery of core APIs and platform stability.
  • OpenAI accepted the performance penalty, betting that future Codex/GPT‑based tooling would ease a later migration.

Managing Tail Latency in Python

  • Asyncio scheduling delay dominated p99+ latency because CPU‑heavy tasks (routing, compression, encryption, health checks) blocked the event loop.
  • Monitoring the event‑loop delay and limiting concurrent requests per process forced massive horizontal scaling of worker processes.

Feature‑Flag Configuration Bottleneck

  • Periodic JSON parsing of a large Statsig config caused all workers in a pod to stall each minute.
  • Fix: reduce config size, increase refresh interval, add jitter to background tasks.

Connection‑Pool Load‑Balancing Issues

  • Default LIFO reuse in aiohttp’s TCPConnector created a metastable feedback loop: slower pods returned connections later, causing subsequent requests to be routed back to them, amplifying overload.
  • Switching to FIFO reuse broke the loop and reduced variance in per‑process utilization.
  • Envoy/Istio now provide connection pooling, HTTP/2 multiplexing, and centralized rate‑limiting to avoid thundering‑herd effects.

Constrained NoSQL API as a Scalability Lever

Habitat deliberately offers a simple NoSQL API rather than arbitrary SQL, keeping request cost predictable. Clients define objects and edges (inspired by TAO) but cannot issue unbounded queries, preventing expensive table scans and protecting the service from misuse. Complex analytical queries are off‑loaded to Rockset via CDC streams.

Migration from Python to Rust

In Q2 2026, OpenAI rewrote Habitat in Rust, now handling 95 % of production traffic. The Rust service is reported to be 6× more CPU‑efficient and 15× more memory‑efficient, with markedly lower average and tail latencies. The Python implementation, which peaked at >20 M requests / s, will be deprecated in the coming weeks.

Future Directions (Part II)

  • Part II will detail multi‑tenancy reliability, read‑performance layering, and the partnership with Azure Cosmos DB that underpins the 500 PB storage layer.
  • Ongoing work includes further optimizations of connection fan‑in, circuit‑breaking, and scaling the Rust service to accommodate continued user growth beyond the current 10× YoY trajectory.

Key Takeaways

  • Habitat’s evolution illustrates how a fast‑moving product org can prioritize developer velocity (Python service) while planning for long‑term efficiency (Rust rewrite).
  • Tight control over API surface, careful event‑loop monitoring, and proper connection‑pool policies are critical when scaling a high‑throughput storage service.
  • Centralizing storage logic enables unified security, observability, and rapid feature rollout across a massive, globally distributed user base.

Sources