komikndr/raylight

What it solves

Raylight provides a way to run large AI models (like Wan, Flux, and Hunyuan Video) across multiple GPUs in ComfyUI. It solves the problem of limited VRAM on single GPUs, allowing users to combine the memory of multiple smaller GPUs to run models that would otherwise cause "Out of Memory" (OOM) errors.

How it works

Raylight uses the Ray framework to manage GPU workers and implements three primary types of parallelism:

• Unified Sequence Parallelism (USP): Splits the image or video sequence tensors across GPU ranks using the XDiT library.
• Fully Sharded Data Parallel (FSDP): Shards model weights across GPUs so that each GPU doesn't need to load the full model, effectively combining available VRAM.
• CFG Parallelism: Splits workload at the Classifier-Free Guidance level.

It also supports CPU offloading for FSDP to further reduce VRAM requirements for very low-memory systems.

Who it’s for

Users of ComfyUI who have multiple GPUs and want to increase their generation speed or run larger models that exceed the capacity of a single card (e.g., using two RTX 4070s instead of one RTX 5090).

Highlights

• Multi-GPU Support: Combines VRAM across multiple cards to enable large model inference.
• Broad Model Support: Compatible with Wan, Flux, Hunyuan Video, Qwen, Chroma, and others.
• Flexible Parallelism: Supports USP, FSDP, DP, and CFG parallelism, including an experimental "Unified Parallel" mode.
• Hardware Compatibility: Tested on NVIDIA (Volta through Blackwell), AMD (MI300X, MI210), and Intel Arc Pro B60.
• Optimization: Supports various attention backends including SageAttn and FlashAttention 2.