Scaling Git at OpenAI: Introducing Pando

OpenAI has developed Pando, a new horizontally scalable Git backend designed to support high-concurrency development by decoupling repository-wide consistency and reusing packfiles. By moving away from traditional three-phase commit and write-ahead log patterns, Pando has reduced median push latency by nearly 80% and now handles nearly a petabyte of Git data per day, all while maintaining full compatibility with standard Git clients.

The Motivation for Pando

The adoption of Codex at OpenAI significantly increased the volume of concurrent branches, pushes, and fetches, pushing existing source control infrastructure to its limits. Traditional Git servers, which grew out of a history of email-based patch exchange, struggled with the scale of modern automated workflows. OpenAI faced frequent slow fetches, failed merges, and reliability issues, necessitating a system that could handle massive parallel writes and high-throughput reads without disrupting existing engineering workflows.

Architectural Principles

Pando achieves scalability by relaxing traditional constraints that typically limit Git performance:

• Causal Consistency: Instead of enforcing repository-wide consistency, which requires global coordination, Pando uses causal consistency. This allows pushes to unrelated branches to proceed in parallel. Clients see a serializable history for individual references, though they may observe different points in time across the repository as a whole.
• Packfile Reuse: Traditional Git servers often generate unique packfiles for each client based on their local history. Pando stores incoming packfiles as immutable blobs and serves them directly to readers. This eliminates the need for expensive on-the-fly computation and allows the system to saturate network bandwidth by serving the same data to many clients concurrently.

Technical Implementation

Pando operates through three primary services: pushes, fetches, and background compaction.

Scaling Pushes

When a client pushes, Pando validates the packfile in a single pass and streams it to a horizontally scalable blob store. It then records a "ref version" in a metadata store using a multi-key transaction. By limiting the set of advertised references, Pando ensures that clients generate packfiles that are more easily shared and reused by other users, keeping the advertisement size constant regardless of the number of branches.

Line-Rate Fetching

Fetch operations are optimized by mapping the Git protocol's haves and wants to a sequence of ref versions. Pando uses the packfile-uri feature—which OpenAI helped improve upstream—to return links to cached packfiles. This allows clients to download data from regional caches, achieving throughput that saturates 10 Gbps network links.

Compaction

To prevent the overhead of managing thousands of small packfiles, Pando employs a background compaction process similar to log-structured merge-trees. This process groups packfiles into geometric "covers" (L0 through L4). These covers improve compression ratios and enable the system to perform fetch planning in logarithmic time, ensuring performance remains stable as the repository grows.

Development and Deployment

Pando was built in 10 weeks using Rust, with development heavily assisted by AI agents. The team utilized formal verification methods, including Lean and TLA+, to prove system properties and identify edge cases. The system was deployed by initially acting as a read/write-through proxy, allowing for a seamless migration of existing workflows without downtime.

OpenAI continues to contribute improvements back to the upstream Git project, particularly in areas related to packfile-uri performance, concurrent downloads, and better tracing for fetch operations. The team is currently exploring ways to share the Pando technology with the broader developer community.