Scaling PostgreSQL to power 800 million ChatGPT users
OpenAI has scaled PostgreSQL to support 800 million users and millions of queries per second (QPS), demonstrating that a single-primary architecture can reliably handle massive read-heavy workloads when paired with rigorous engineering optimizations. By utilizing a single Azure PostgreSQL flexible server instance as the primary and deploying nearly 50 read replicas across multiple global regions, OpenAI has maintained low double-digit millisecond p99 client-side latency and "five-nines" availability.
Overcoming Single-Primary Limitations
While a single-primary architecture is efficient for read-heavy traffic, it creates a bottleneck for write operations and a potential single point of failure. OpenAI addressed these challenges through the following strategies:
Write Load Mitigation
To prevent the primary instance from being overloaded during write spikes, OpenAI migrated shardable, write-heavy workloads to sharded systems like Azure Cosmos DB. The organization now prohibits adding new tables to the existing PostgreSQL deployment, defaulting all new workloads to sharded systems. Additionally, application-level optimizations—such as fixing redundant writes and introducing lazy writes—were implemented to smooth traffic spikes.
High Availability and Failure Mitigation
To eliminate the primary as a total single point of failure, OpenAI offloaded critical read queries to replicas. This ensures that read-only requests remain available even if the primary instance fails. For full recovery, the primary is run in High-Availability (HA) mode with a hot standby—a synchronized replica ready for immediate promotion to minimize downtime.
Technical Optimizations for High QPS
OpenAI implemented several architectural layers to protect PostgreSQL from resource exhaustion and performance degradation.
Connection Pooling and Latency
Using PgBouncer as a proxy layer in statement or transaction pooling mode, OpenAI reduced the number of active client connections and lowered average connection time from 50ms to 5ms. To minimize network overhead, PgBouncer pods are co-located with clients and replicas within the same region.
Caching and "Cache-Miss Storms"
To protect the database from sudden surges in read traffic caused by cache misses, OpenAI implemented a cache locking (leasing) mechanism. This ensures that only one reader fetching a specific missing key can query PostgreSQL to repopulate the cache, forcing other concurrent requests for the same key to wait rather than hitting the database simultaneously.
Query and Schema Management
OpenAI optimizes queries to avoid Online Transaction Processing (OLTP) anti-patterns, specifically avoiding complex multi-table joins. In one instance, a query joining 12 tables was identified as a cause of high-severity incidents. The team now moves complex join logic to the application layer and uses idle_in_transaction_session_timeout to prevent long-running idle queries from blocking autovacuum.
Schema changes are strictly controlled: only lightweight operations that do not trigger full table rewrites are permitted, and a strict 5-second timeout is enforced on all schema changes.
Scaling Read Replicas and Workload Isolation
To maintain global performance, OpenAI uses nearly 50 read replicas. However, because the primary must stream Write Ahead Log (WAL) data to every replica, scaling further could eventually overload the primary's CPU and network bandwidth.
Cascading Replication
OpenAI is collaborating with the Azure PostgreSQL team to implement cascading replication, where intermediate replicas relay WAL data to downstream replicas. This is intended to allow scaling to over 100 replicas without overwhelming the primary instance.
Workload Isolation
To prevent "noisy neighbor" problems, OpenAI isolates workloads by splitting requests into low-priority and high-priority tiers, routing them to separate instances. This ensures that resource-intensive low-priority requests do not degrade the performance of critical high-priority features.
Performance Results
Through these optimizations, OpenAI has achieved the following production metrics:
- Availability: Five-nines availability.
- Latency: Low double-digit millisecond p99 client-side latency.
- Reliability: Only one SEV-0 PostgreSQL incident in the last 12 months, which occurred during the launch of ChatGPT ImageGen when write traffic surged by over 10x.