The Economics of Load-Balanced Systems and Erlang C

Increasing the number of servers in a load-balanced system while maintaining constant per-server utilization reduces mean latency and improves tail percentiles. In a system where the load increases linearly with the number of servers, the time spent in the queue asymptotically approaches zero as the system scales, meaning larger clusters provide better latency at the same utilization levels.

Latency Behavior in M/M/c Queuing Systems

In an M/M/c queuing system—characterized by a Poisson arrival process, exponentially distributed service times, and c backend servers—the relationship between scale and latency is non-linear. When a load balancer with an infinite queue distributes traffic to c servers, each capable of handling one concurrent request, the client-observed mean request time decreases as c increases, even if the per-server load remains constant (e.g., at 80% utilization).

This behavior is explained by Erlang's C formula, which calculates the probability that an incoming request must be enqueued rather than handled immediately. As the number of servers increases, the probability of a request being enqueued drops significantly:

• Small Scale: A system with fewer servers at a specific utilization rate has a higher probability of requests hitting the queue.
• Large Scale: A system with more servers at the same utilization rate is more efficient at absorbing bursts of traffic, resulting in a lower probability of queuing.

Impact on Mean and Tail Latency

While mean latency is often a controversial metric, Monte-Carlo simulations demonstrate that this latency improvement is not limited to the average. The median (p50) and high percentiles (p99 and p99.9) follow a similar downward trend as the number of servers increases. This indicates that scaling the number of servers effectively reduces tail latency without introducing hidden performance regressions.

Economic Implications for Cloud Services

Scaling the number of servers provides a distinct economic advantage: operators can achieve either lower latency for the same resource utilization or higher utilization for the same latency target. Because most of these latency gains occur at relatively modest values of c, this benefit is applicable to both small and medium-sized services, not just hyperscale infrastructure.

Model Assumptions and Constraints

The M/M/c model relies on specific assumptions that may not always align with real-world production environments:

• Arrival Process: Assumes a Poisson arrival process.
• Service Time: Assumes exponential service time. While realistic services often exhibit log-normal service time distributions, the general trend of latency improvement with scale typically persists.
• Stability Requirement: For the system to remain stable and avoid an infinite queue, the ratio of the mean arrival rate ($λ$) to the product of the number of servers ($c$) and the mean service rate ($μ$) must be less than 1 ($λ/cμ < 1$). If the arrival rate exceeds the system's total processing capacity, latency will grow without bound.