Cloudflare Meerkat: Globally Distributed Consensus via QuePaxa

Cloudflare Meerkat provides high-availability global consensus by eliminating the "tyranny of timeouts" found in leader-based protocols.

Cloudflare has introduced Meerkat, an experimental distributed consensus service designed to manage critical control-plane state across 330+ global data centers. Unlike traditional consensus systems that rely on a single authoritative leader, Meerkat utilizes the QuePaxa algorithm, allowing any replica to drive consensus. This architecture ensures that the system remains available for reads and writes even if the current leader fails or network latency fluctuates wildly, preventing the service interruptions Cloudflare previously experienced with Raft-based systems.

Requirements for Global Control-Plane Data

Control-plane data—such as resource placement information or database leadership status—requires a combination of strong consistency and high fault tolerance to operate reliably across a wide-area network (WAN).

Linearizability (Strong Consistency)

Meerkat aims for linearizability, the strongest consistency model. In a linearizable system, all operations are ordered exactly as they occur in real time. This ensures that any read performed after a write is guaranteed to see that write, regardless of which global replica the client contacts. This model simplifies development by allowing programmers to reason about distributed state as if it were local memory on a single-threaded machine.

Fault Tolerance and Correctness

Meerkat is designed to maintain correctness and availability under the following conditions:

• Majority Availability: The system remains available for reads and writes as long as a majority of machines (f+1 in a system of 2f+1) are alive and communicating.
• Client Access: A client can interact with any machine that is connected to a majority of live machines.
• Non-Byzantine Faults: The system remains correct provided no actor is actively malicious, though it handles machine crashes, restarts, and network degradation.

Architecture: The Meerkat Log and QuePaxa

Meerkat functions as a foundation upon which applications, such as a transactional key-value (KV) store or a leasing system, are built.

The Distributed Log

At its core, Meerkat maintains a sequence of "slots." Each slot can either be empty or contain a decided event.

• Consistency via Slots: If two replicas decide on a value for a specific slot, those values must be identical.
• Linearizing Reads: To ensure linearizability, even get requests create log events. If a replica attempts to read from a slot it believes is empty, but a majority has already decided a value for that slot, the replica is forced to synchronize with the majority before completing the read, effectively linearizing the read after the preceding write.

Why QuePaxa Over Raft?

Cloudflare transitioned from Raft to QuePaxa to solve specific availability issues inherent in wide-area networks:

1. Elimination of Leader Bottlenecks: In Raft, all writes must go through a single leader. If the leader fails or slows down, the entire system blocks. In QuePaxa, any replica can drive consensus for the latest slot.
2. Avoiding Timeout Issues: Raft relies on timeouts to trigger new leader elections. In unpredictable WANs, timeouts that are too short cause constant unnecessary elections, while timeouts that are too long increase downtime. QuePaxa does not halt progress due to timeouts.
3. Constructive Interference: While Raft's leadership campaigns can interfere and block writes, QuePaxa allows multiple replicas to propose values concurrently; these proposals work together to decide a value rather than destructively interfering.

Performance Trade-offs and Optimizations

Because consensus requires multiple network round-trips, Meerkat is not intended as a general-purpose database but rather for infrequently written control-plane data.

Latency Constraints

Decision latency is proportional to the network latency between a majority of replicas. A proposal typically takes one round trip if driven by a leader and three or more if driven by a non-leader.

Performance Mitigations

To improve throughput and reduce perceived latency, Meerkat employs several strategies:

• Batching: Replicas can group multiple writes into a single proposal.
• Replica Placement: Developers can place replicas closer together to reduce round-trip times for services that do not require full global distribution.
• Stale Reads: For applications that do not require strict linearizability, reads can be served from a replica's local data without triggering a consensus round.
• Bundled Operations: The KV store supports general transactions and compare-and-swap (CAS) writes within a single consensus round.

Current Status

Meerkat is currently an internal experimental service. Proof-of-concept deployments with up to 50 global replicas have demonstrated that the cluster continues to operate without an increase in error rates even when leaders fail constantly.