tensorforger/FluxRT

What it solves

FluxRT is a real-time stream editing pipeline designed to transform webcam or video feeds with low latency. It enables interactive prompt updates and reference image conditioning, making it suitable for AI VTubing, virtual try-ons, and creative coding, while remaining optimized for consumer-grade GPUs.

How it works

FluxRT uses the FLUX.2-Klein instruct image editing model and employs several optimizations to achieve real-time performance:

• Spatial KV Cache: A custom cache variant that reuses intermediate computations from previous frames. Since video frames are temporally coherent, it only recomputes 20-50% of tokens per frame, reducing GPU utilization and latency.
• Real-Time Frame Interpolation: Integrates the RIFE model to generate intermediate frames, ensuring smooth motion without increasing the core model's latency.
• Multiprocessing & Shared Memory: Decouples computation, I/O, and rendering into separate processes (Main, Inference, and Output Scheduler) using shared memory for near-zero-copy frame transfers.
• Model Compilation: Uses TorchInductor to maximize runtime performance.

Who it’s for

Content creators, AI VTubers, and developers interested in real-time AI image transformation and interactive visual arts.

Highlights

• High Performance: Achieves 15-40 FPS on RTX 4090/5090 GPUs with end-to-end latency around 0.2-0.3 seconds.
• Virtual Webcam Support: Includes a GUI that can create a virtual webcam, allowing the output to be streamed into apps like OBS, Zoom, and Chrome.
• Reference Image Conditioning: Natively supports FLUX.2 reference image features for tasks like real-time AI fitting rooms.
• Extensibility: Supports optional extensions like TAEF2 for faster decoding, Flow Upscaler for super-resolution, int8 quantization for lower VRAM usage, and LivePortrait for real-time lip transfer.