Hugging Face Remote VAEs for Inference Endpoints
Hugging Face has introduced an experimental pilot to delegate the Variational Autoencoder (VAE) decoding process to remote endpoints. This approach allows users to perform high-resolution image and video synthesis on consumer GPUs by offloading the memory-intensive decoding stage to an Inference Endpoint, thereby avoiding the VRAM bottlenecks associated with local VAE execution.
Solving VRAM Bottlenecks in Latent-Space Diffusion
In latent-space diffusion models, the VAE decoder is often the most memory-intensive component during the final stage of generation. For users on consumer-grade hardware, this creates a significant barrier to high-resolution output. Hugging Face identifies two common local workarounds and their respective drawbacks:
- Offloading: Moving the VAE to CPU RAM creates device transfer overhead, which increases overall inference latency.
- Tiling: Processing the image in smaller tiles reduces memory usage but can negatively impact the final image quality.
By delegating the decoding process to a remote endpoint, users can maintain high image quality and reduce local VRAM consumption without the latency penalties of local offloading.
Technical Implementation and Integration
This experimental feature is developed by the Diffusers team and utilizes huggingface-inference-toolkit with custom handlers. The implementation is open source, and no data is stored or tracked during the process.
Integration with Diffusers
To use remote VAEs, users must install the diffusers library from the main branch:
pip install git+https://github.com/huggingface/diffusers@main
Decoding is handled via the remote_decode helper method from diffusers.utils.remote_utils.
Model Support and Endpoints
Hugging Face provides dedicated endpoints for several popular models:
| Model | VAE Endpoint |
|---|---|
| Stable Diffusion v1 | https://q1bj3bpq6kzilnsu.us-east-1.aws.endpoints.huggingface.cloud |
| Stable Diffusion XL | https://x2dmsqunjd6k9prw.us-east-1.aws.endpoints.huggingface.cloud |
| Flux | https://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud |
| HunyuanVideo | https://o7ywnmrahorts457.us-east-1.aws.endpoints.huggingface.cloud |
Capabilities and Use Cases
Image and Video Generation
The remote_decode function supports various modalities and model-specific requirements. For example, Flux requires height and width parameters because its latents are packed. HunyuanVideo supports outputting results directly as .mp4 files.
Concurrency via Queueing
Because the decoding happens remotely, users can implement a queueing system to improve concurrency. While a remote endpoint is decoding the current latent, the local GPU can continue generating the next latent in the pipeline. This allows for a streamlined generation-decoding loop that maximizes hardware utilization.
Performance and Memory Impact
Local VAE decoding requirements scale sharply with resolution and GPU capacity. For Stable Diffusion v1.5 at 1024x1024 resolution, memory usage can reach 56.3% on an RTX 3070, whereas tiled decoding reduces this to 16% but increases inference time by 77%.
For SDXL at 1024x1024, memory consumption is even more severe, reaching 93% on an RTX 3080 and 96.4% on an RTX 3070, making remote decoding a viable alternative for users who cannot fit the full VAE process in VRAM.