Efficient Request Queueing for LLM Performance Optimization

Serving Large Language Models (LLMs) to multiple users in parallel is challenging because requests compete for limited GPU resources. To maintain a consistent user experience, organizations must implement sophisticated queueing and scheduling strategies that prevent a single high-volume user from monopolizing the inference engine.

The Problem: Resource Blocking by Power Users

In a standard inference engine (such as vLLM or HuggingFace TGI), requests are typically handled by a worker, a queue, and a scheduler. Because GPU calculations are more efficient when performed in batches, the backend queue allows the scheduler to group multiple requests into a single batch.

However, when a "power user" sends a large volume of requests, they can fill the backend queue. This creates a blocking effect where subsequent requests from other users must wait until all the power user's requests are processed, regardless of the urgency or volume of the new requests.

Solution: Implementing Fair Scheduling

To prevent resource blocking, TNG implements an "LLM-Server" (an API server) that sits between the users and the inference backend. Instead of sending requests directly to the backend, the LLM-Server manages separate queues for each user and model.

Round-Robin Scheduling

The LLM-Server employs a round-robin scheduler rather than a First-In-First-Out (FIFO) approach. This ensures that requests from different users are prioritized, allowing a user with a single request to be served more quickly even if another user has multiple requests pending in their specific queue.

Potential Scheduling Extensions

Beyond simple request counts, scheduling can be further refined based on several metrics:

  • Processing Time: Prioritizing shorter requests to reduce overall wait times, although estimating generation length is difficult.
  • Cache Optimization: Arranging requests by similarity to maximize KV-cache hits, a technique used by frameworks like NVIDIA Dynamo and AIBrix.
  • Business Cost: Prioritizing requests based on the financial cost of the operation.
  • Priority Tiers: Establishing different queues for different use cases. For example, interactive chat interfaces receive high priority to maintain a responsive UI, while batch API jobs for code reviews or benchmarks receive low priority.

Managing Backend Backpressure

Fair scheduling at the LLM-Server level is ineffective if the server sends all prioritized requests to the backend immediately, as they would simply accumulate in the backend's FIFO queue and recreate the blocking problem.

Metric-Based Rate Adjustment

Because some backends (like vLLM) do not allow limiting the maximum number of elements in the backend queue, the LLM-Server must dynamically adjust the rate at which it forwards requests. TNG achieves this by fetching Prometheus metrics from the vLLM /metrics endpoint.

By monitoring the backend queue length metric, the LLM-Server only forwards new requests when the queue length is below a specific threshold (e.g., three). This minimizes latency for new users while balancing the trade-off between low latency and GPU underutilization.

Advanced Metric Feedback Loops

This feedback loop allows for further optimizations based on real-time performance:

  • Token Speed Thresholds: If the time-per-output-token metric exceeds a certain limit (e.g., 150ms), the server can stop scheduling new requests to maintain a minimum generation speed (e.g., >7 tokens/s).
  • Priority-Specific Thresholds: Low-priority batch requests may only be scheduled when the backend queue is completely empty to ensure they never increase latency for high-priority users.

Alternative: Backend-Side Priority Scheduling

Recent versions of vLLM have introduced priority-based scheduling, allowing requests to be tagged with a priority level. High-priority requests can "jump" the queue and even move directly into the processed batch, potentially evicting lower-priority requests back into the waiting queue.

Comparison with LLM-Server Scheduling

While backend-side priority scheduling simplifies some logic, an upstream LLM-Server remains necessary for several reasons:

  1. Compatibility: Backend-priority features are available in vLLM but not in HuggingFace TGI.
  2. Rate Control: Backend scheduling does not control the rate of request submission based on performance metrics like time-per-output-token.
  3. Priority Assignment: An objective upstream instance is required to assign priorities to requests based on user identity or application type.

Summary of Queueing Strategies

Strategy Implementation Location Primary Benefit Limitation
FIFO Backend Simple implementation Power users block others
Fair Scheduling LLM-Server Prevents user blocking Requires upstream server
Metric-Based Backpressure LLM-Server Minimizes backend latency Requires metric polling
Priority Scheduling Backend (vLLM) Immediate high-priority processing No rate control; vLLM only

Sources