Scaling Git at OpenAI: How Pando Achieves Petabyte‑Scale Throughput

TL;DR

OpenAI built Pando, a custom Git backend that keeps the standard Git client but delivers 5× lower push latency and petabyte‑scale daily data transfer by relaxing repository‑wide consistency and reusing packfiles across clients.

Why OpenAI Needed a New Git Backend

  • Codex enabled many engineers to work in parallel, exploding the number of branches, pushes, merges, and fetches.
  • By late 2023 the monorepo suffered slow pushes, failed merges, and reliability below 90 %.
  • Existing scaling approaches (three‑phase commit or a shared write‑ahead log) still required global coordination or a single manifest, creating bottlenecks for concurrent writes or reads.
  • OpenAI wanted to retain Git‑compatible workflows so improvements could benefit the broader community.

"We saw a way to keep that familiar workflow while rebuilding the backend to handle pushes to different branches in parallel and serve the same history to many readers concurrently." – OpenAI blog

Core Design Goals of Pando

  1. Horizontally scalable writes – allow many branches to be updated simultaneously without a global lock.
  2. Maximum read throughput – serve the same packfile to many clients, saturating network bandwidth.
  3. Git‑protocol compatibility – no changes required to existing client tooling.

Architectural Breakthroughs

1. Causal Consistency Instead of Global Consistency

  • Each branch update creates an immutable ref version that points to the uploaded packfile and its dependencies.
  • Writes are serializable per‑ref; only pushes to the same branch conflict.
  • Readers see a causally consistent view: different branches may reflect different points in time, which matches real‑world usage where developers care about the order of updates to a single ref, not a global snapshot.

2. Packfile Reuse Across Clients

  • Pushes are stored as immutable blobs in a scalable object store.
  • Fetches map a client’s have/want sets to the stored packfiles that already contain the needed objects.
  • The Git protocol’s packfile‑uri extension lets the server return URLs to multiple pre‑existing packs, avoiding on‑the‑fly pack generation.
  • Result: read‑side CPU and I/O are reduced dramatically; a single pack can satisfy thousands of fetches.

3. Layered Compaction (L0‑L4)

  • Every push creates an L0 pack (the smallest unit).
  • Background workers merge packs into larger covers (L1‑L4) using a log‑structured‑merge‑tree style algorithm.
  • Covers act as a skip‑list for fetch planning, enabling log‑Δ metadata lookups where Δ is the distance between the client’s current tip and the desired tip.
  • Compaction improves compression ratios and reduces the number of pack URIs a client must download.

Comparison with Conventional Approaches

Approach Write Bottleneck Read Bottleneck Consistency Model
Three‑phase commit (3PC) Global quorum across replicas; pushes to unrelated branches serialize. Reads are fast (local replica) but replica disks saturate under heavy write load. Strong, point‑in‑time snapshot of all refs.
Write‑ahead log (WAL) Single manifest update; compare‑and‑swap latency limits push rate. Servers must read and index packs from the blob store, hurting fetch latency. Strong ordering of all ref updates.
Pando Per‑ref atomic compare‑and‑swap; only the ref pointer is contended. Packfile reuse and cache‑local delivery keep fetch latency low even at petabyte scale. Causal consistency – serializable per ref, no global snapshot.

In simulations with 1 000 concurrent writers, Pando maintained low push‑to‑fetch latency while 3PC suffered write stalls and WAL suffered read stalls. Real‑world production confirms thousands of concurrent pushes with median latency reduced by ~80 %.

Push Flow in Pando

  1. Client creates a packfile and runs git push.
  2. Pando streams the pack to the blob store while validating it in a single pass.
  3. A multi‑key transaction records a new ref version (metadata) and atomically updates the ref pointer.
  4. Synchronous metadata writes guarantee read‑your‑own‑write consistency; asynchronous work (e.g., scheduling compaction) proceeds in the background.

Complexity: O(1) metadata reads/writes regardless of push size.

Fetch Flow in Pando

  1. Client sends have and want IDs.
  2. Pando maps these to a set of ref versions and selects the minimal covering packfiles.
  3. The server returns an empty inline pack plus packfile‑uri links to the selected packs.
  4. Client downloads packs directly from regional caches (Express) – cache hit rates > 99 %.

Complexity: O(log Δ) metadata reads, where Δ is the number of ref versions between client state and target.

Performance Numbers

  • Median push latency: ~5× lower than the previous system (≈80 % reduction).
  • Daily data served: ~1 PB of Git objects.
  • Peak throughput: > 57 TB of packs transferred in a single hour, > 99 % served from local disk cache.
  • Network utilization: Clone/fetch operations saturate 10 Gbps links on commodity VMs.

Deployment Experience

  • Pando was built in 10 weeks using Rust, with the core design authored by a single engineer and implementation assisted by Codex‑generated code.
  • Initial rollout used a read‑through proxy to keep the legacy system in sync, enabling a zero‑downtime migration.
  • An internal Git wrapper phased the migration for developers, CI, and third‑party integrations.
  • Codex‑driven automation helped developers migrate their environments with a simple command.

Contributions Back to Git

  • Added upstream support for packfile‑uri and improved concurrent download handling.
  • Implemented “explicit haves” to let servers advertise a branch tip separately from the history a push may omit.
  • Added tracing for packfile downloads and byte‑count metrics, aiding performance debugging.
  • All patches are available in OpenAI’s public Git fork: https://github.com/openai/git and tracked at https://oai-git.openai.chatgpt.site/.

Community Context

OpenAI’s effort follows a wave of large‑scale Git engineering posts, including:

  • Cursor’s Git at Any Scale
  • GitHub’s Architecture Optimization series
  • Anthropic’s Redesigning Source Control
  • East River Source Control’s What Comes After Git

These works share the same problem space—massive monorepos, AI‑generated code, and the need for high‑throughput source control—but differ in implementation details (e.g., Google‑style monorepo services vs. Pando’s causal consistency).

Takeaways

  • Pando proves that you can keep the Git client unchanged while radically redesigning the backend to handle petabyte‑scale traffic.
  • Causal consistency is sufficient for most developer workflows and eliminates the need for a global ordering point.
  • Packfile reuse and layered compaction turn fetches into a cache‑hit problem rather than a CPU‑intensive pack generation task.
  • The design scales horizontally on commodity hardware, making it a viable blueprint for any organization facing similar monorepo growth.

What’s Next

OpenAI plans to open‑source parts of Pando, explore broader agent‑centric use cases, and continue upstream contributions to Git to benefit the entire ecosystem.

Sources