Hugging Face ZeroGPU Ahead-of-Time Compilation Guide

Hugging Face has introduced ahead-of-time (AoT) compilation for ZeroGPU Spaces, allowing developers to optimize model latency and eliminate cold-start timings. By compiling models once and reloading them instantly across short-lived GPU processes, users can achieve speedups ranging from 1.3Ñ–1.8Ñ on models such as Flux, Wan, and LTX.

ZeroGPU Architecture and the Need for AoT

ZeroGPU utilizes a just-in-time approach to GPU initialization to maximize resource efficiency. Instead of reserving a GPU for the entire lifetime of a Space, ZeroGPU forks the process, executes GPU tasks on an Nvidia H200 (currently using the 3g.71gb MIG slice), and kills the fork once the task is complete.

While torch.compile is effective in standard environments, it relies on a filesystem cache on ZeroGPU because processes are spun up freshly for almost every task. This caching process can take from a few dozen seconds to several minutes, making it impractical for real-time demos. AoT compilation solves this by exporting a compiled model once, which can then be reloaded instantly in any process.

Implementing AoT Compilation on ZeroGPU

Implementing AoT compilation involves a five-step workflow using the spaces package and PyTorch utilities:

  1. Capture Example Inputs: Use spaces.aoti_capture to intercept the arguments and keyword arguments passed to a model component (e.g., the transformer in a diffusion pipeline).
  2. Export the Model: Use torch.export.export to convert the model into a ExportedProgram, which is a computation graph containing tensor computations and model parameters.
  3. Compile the Exported Model: Use spaces.aoti_compile (a wrapper around torch._inductor.aot_compile) to generate an AoT-compiled binary.
  4. Apply the Compiled Model: Use spaces.aoti_apply to patch the model's forward method and remove original model parameters from memory to prevent Out-of-Memory (OOM) errors.
  5. Wrap in GPU Context: Because compilation is hardware-dependent and requires a real GPU, the compilation steps must be wrapped in a @spaces.GPU function during the app's startup phase.

For the FLUX.1-dev model, this process resulted in a 1.7x speedup.

Advanced Optimization Techniques

FP8 Quantization

AoT compilation can be combined with FP8 post-training dynamic quantization via the torchao library. Since ZeroGPU uses H200 GPUs (compute capability 9.0+), it supports FP8, which provides an additional 1.2x speedup.

Handling Dynamic Shapes

To support varying image or video resolutions, developers can define dynamic dimensions using torch.export.Dim. For Flux.1-Dev, this involves making the flattened_latent_dim of hidden_states and the height * width of img_ids dynamic. These configurations are passed to torch.export.export via a dynamic_shapes map.

Multi-Compile and Shared Weights

When dynamism is too extreme for single-graph dynamic shapes (e.g., in the Wan video generation family), developers can compile one model per resolution while sharing model parameters and dispatching the correct compiled graph at runtime.

FlashAttention-3 (FA3)

ZeroGPU is compatible with FlashAttention-3. To avoid the time-consuming process of building FA3 from source, Hugging Face provides the kernels library, which allows users to load pre-built, hardware-compatible kernels (e.g., kernels-community/vllm-flash-attn3).

Regional Compilation

Instead of compiling the entire model, developers can compile only repeated blocks of computation (e.g., FluxTransformerBlock and FluxSingleTransformerBlock in Flux). This significantly reduces cold start times—reducing compilation time for Flux.1-Dev from 6 minutes to 30 seconds—while maintaining the same speedups as full model compilation.

Deployment and Distribution

Compiled graph modules can be serialized as artifacts and uploaded to the Hugging Face Hub. By saving the compiled model graph without the model parameters, storage remains light. Demos can then download and load these pre-compiled graphs to skip the compilation phase entirely during startup.

Sources