vLLM Multi-LoRA Serving for MoE Models
vLLM has introduced an efficient solution for Multi-Low-Rank Adaptation (Multi-LoRA) serving, allowing organizations to host dozens of fine-tuned models on a single GPU. This eliminates the need for dedicated compute endpoints for every custom model, which often leads to idle GPU capacity when individual models do not receive enough traffic to saturate their hardware.
Multi-LoRA Serving for MoE Models
Multi-LoRA allows multiple custom models to share the same GPU by keeping the original base model weights frozen and swapping small, trainable adapters in and out per request. This is particularly beneficial for Mixture of Experts (MoE) models, where a router directs input tokens to specialized neural networks (experts).
In an MoE architecture, each expert uses an "expand-then-compress" pattern: a gate_up projection expands the hidden state into a larger intermediate space, and a down projection compresses it back. When LoRA is applied, each expert requires four additional kernel operations: a shrink and expand operation for both the gate_up and down projections.
To handle this, vLLM implemented a fused_moe_lora kernel. This kernel integrates LoRA operations directly into the fused_moe kernel, managing the compound sparsity created by both expert routing (tokens assigned to different experts) and adapter selection (requests using different LoRA adapters).
Technical Optimizations for Performance
Initial implementations of multi-LoRA for MoE models suffered from high latency. vLLM and AWS utilized NVIDIA Nsys and NCU to identify and resolve bottlenecks through three primary optimization layers:
Execution Optimizations
Profiling revealed that the Triton compiler was treating input-length-dependent variables as compile-time constants, causing the fused_moe_lora kernel to be recompiled for every new context length. This resulted in a 10× Time To First Token (TTFT) regression compared to the base model. This was resolved by adding a do_not_specialize compiler hint, ensuring the kernel is compiled once and reused across all context lengths.
Kernel-Level Optimizations
Because LoRA matrices are "skinny" (where the rank r is significantly smaller than the hidden state dimension), standard GEMM kernels perform poorly. The following optimizations were implemented:
- Split-K Work Decomposition: This strategy splits the summation over the inner dimension
Kacross multiple thread groups, computing partial sums in parallel to improve load balancing for skinny matrices. Atomic additions were optimized usingsem="relaxed"in the Triton compiler. - CTA Swizzling: Cooperative Thread Array (CTA) swizzling reorders the GPU schedule so that thread groups working on nearby columns run simultaneously, increasing L2 cache reuse.
- Masking Reduction: An
EVEN_Kparameter was introduced to skip conditional masking checks when matrix dimensions divide evenly into block sizes, reducing overhead on every load operation. - Kernel Fusion: The addition of LoRA weights to base model weights was fused into the LoRA expand kernel to reduce kernel launch overhead.
Amazon-Specific Tuning
Default Triton kernel configurations optimized for standard fused MoE did not account for the compound sparsity of multi-LoRA. AWS developed custom tuned configurations for the four fused_moe_lora operations (gate_up_shrink, gate_up_expand, down_shrink, down_expand). These configurations are automatically loaded for customers using Amazon SageMaker AI and Amazon Bedrock.
Performance Benchmarks and Availability
These optimizations significantly improved performance for MoE models like GPT-OSS 20B, Qwen3-MoE, DeepSeek, and Llama MoE, as well as dense models such as Llama 3.3 70B and Qwen3 32B.
For GPT-OSS 20B, the transition from vLLM 0.11.1rc3 to vLLM 0.15.0 resulted in a 454% improvement in Output Tokens Per Second (OTPS) and an 87% reduction in TTFT. Further optimizations available on Amazon SageMaker AI and Amazon Bedrock provide an additional 19% increase in OTPS and an 8% reduction in TTFT compared to vLLM 0.15.0.
Summary of GPT-OSS 20B Results:
| Version/Platform | OTPS | TTFT |
|---|---|---|
| vLLM 0.11.1rc3 | 32 (Initial) | ~1.2s (Estimated) |
| vLLM 0.15.0 | 144 | 135 ms |
| Amazon SageMaker/Bedrock | 171 | 124 ms |
These improvements are available in vLLM version 0.15.0 or later.