Unlocking asynchronicity in continuous batching

Synchronous batching leaves the GPU idle a quarter of the time

In synchronous continuous batching the CPU and GPU take turns: while the GPU computes a forward pass the CPU waits, and while the CPU prepares the next batch the GPU waits. Profiling an 8B model generating 8K tokens with batch size 32 shows a total time of 300.6 seconds, of which 24.0% is spent with the GPU idle waiting for the CPU. This idle gap represents a throughput loss that continuous batching alone does not eliminate.

Creating concurrency requires non‑default CUDA streams

To overlap CPU and GPU work we must launch GPU operations without blocking the CPU. In PyTorch operations without an explicit stream run on the default stream, which synchronizes with all other streams and forces the CPU to wait for the GPU to finish. Using non‑default streams lets the CPU enqueue work and regain control immediately. We need three separate streams: one for host‑to‑device (H2D) transfers, one for GPU compute, and one for device‑to‑host (D2H) transfers, because these operations are independent and can run concurrently when placed in different streams.

Enforcing ordering between streams needs CUDA events

Launching the three streams independently causes race conditions: compute may start before the H2D transfer finishes, and D2H may transfer results before compute ends. A CUDA event records a marker in a stream; another stream can wait for that marker before proceeding. By recording an event after each H2D transfer and waiting on it in the compute stream, and similarly recording after compute and waiting in the D2H stream, we create a pipeline where the GPU enforces the correct order while the CPU remains free after enqueuing all work.

Avoiding data corruption and handling carry‑over requires double buffering and a mask

Preparing batch N+1 while batch N is computing risks overwriting input buffers that the GPU is still reading. The solution is to maintain two sets of host‑side and device‑side tensors (slots A and B) and alternate between them, which doubles memory usage but prevents race conditions. Because a request that appears in both batches needs its newly generated token as input for the next batch, we insert a placeholder token (value 0) in the batch N+1 input buffer and later replace it with the actual token from batch N’s output. This replacement, called carry‑over, is performed by a carry‑over mask that specifies destination positions; the mask is applied inside the CUDA graph at negligible cost.

The full asynchronous loop overlaps CPU batch preparation with GPU compute

Step 0 (cold start) dispatches batch 0 using the synchronous path. From step 1 onward the CPU, while the GPU computes batch N in slot A, prepares batch N+1 in slot B: evicting finished requests, admitting new ones, updating the KV cache, and building the carry‑over mask. Once inputs are ready the CPU enqueues the H2D transfer on the H2D stream, records an event, waits on it in the compute stream, launches the forward pass, records another event, and waits on it in the D2H stream before launching the output transfer. The CPU then blocks only on a final synchronization event to copy results back, after which it immediately begins preparing the next batch. This pattern repeats, keeping the GPU busy as long as batch N+1’s inputs are ready when batch N finishes.

Experimental results show a 22% speedup

Running the same workload as the synchronous baseline (8K tokens, batch size 32, 8B model) with the asynchronous pipeline yields a timeline where CPU and GPU overlap almost continuously. The GPU is active for 99.4% of total runtime, up from 76.0% in the synchronous case. Total generation time drops from 300.6 seconds to 234.5 seconds, a speedup of about 22 %. The remaining gap corresponds to the unavoidable synchronization point where the CPU waits for the D2H transfer to complete; no new kernels or model changes are required.

Conclusion

By replacing schedule‑based dependencies with data‑based dependencies using CUDA streams and events, and by solving race conditions with double buffering and carry‑over masks, we decouple CPU and GPU work in continuous batching. This allows the GPU to stay saturated, delivering a substantial throughput improvement for LLM inference while preserving model accuracy.