SDXL and Stable Diffusion Fast Inference with Latent Consistency LoRAs
Hugging Face has introduced Latent Consistency LoRAs (LCM LoRAs), a method that enables Stable Diffusion and SDXL models to generate high-quality images in just 4 to 8 steps, compared to the typical 25 to 50 steps. This advancement significantly reduces inference time, allowing for near real-time image generation on high-end GPUs and drastically improving accessibility on lower-end hardware.
LCM LoRA Method Overview
LCM LoRAs utilize Low-Rank Adaptation (LoRA) to implement latent consistency distillation without requiring the full model to be distilled separately. Instead of a costly and data-intensive full-model distillation process, a small number of adapter layers are trained. These resulting LoRAs can be applied to any fine-tuned version of the base model, providing the benefits of latent consistency distillation across various model versions.
The implementation process involves three primary steps:
- Selecting a teacher model (e.g., SDXL base or a fine-tuned version).
- Training an LCM LoRA using Parameter-Efficient Fine-Tuning (PEFT).
- Applying the LoRA with any SDXL diffusion model and the
LCMSchedulerfor high-quality, low-step inference.
Performance Benchmarks and Hardware Impact
LCM LoRAs reduce image generation time by an order of magnitude across various hardware configurations. For a 1024x1024 image, the speed improvements are as follows:
| Hardware | SDXL LoRA LCM (4 steps) | SDXL standard (25 steps) |
|---|---|---|
| Mac, M1 Max | 6.5s | 64s |
| 2080 Ti | 4.7s | 10.2s |
| 3090 | 1.4s | 7s |
| 4090 | 0.7s | 3.4s |
| T4 (Google Colab Free) | 8.4s | 26.5s |
| A100 (80 GB) | 1.2s | 3.8s |
| Intel i9-10980XE CPU | 29s | 219s |
On an NVIDIA RTX 4090, response times are under one second, enabling the use of SDXL in applications requiring real-time interaction. On M1 Macs, generation time drops from approximately one minute to roughly 6 seconds.
Quality, Guidance, and Model Compatibility
Inference Quality vs. Step Count
While 1-step generation produces only approximate shapes without texture, quality improves rapidly between 4 and 6 steps. In comparison, standard SDXL pipelines typically produce unusable images until approximately 20 steps, with peak detail requiring 50 steps.
Guidance Scale and Negative Prompts
For maximum speed, a guidance_scale of 1 is recommended, which effectively disables guidance. To utilize negative prompts, a guidance scale between 1 and 2 can be used; however, values larger than 2 are generally ineffective.
Compatibility with Fine-tuned Models
LCM LoRAs are compatible with any fine-tuned SDXL or Stable Diffusion model. For example, the technique can be applied to collage-diffusion (a Dreambooth fine-tune of Stable Diffusion v1.5) by loading the corresponding LCM LoRA for SD v1.5, enabling 4-step inference for specialized styles.
Integration and Advanced Workflows
Diffusers Integration
LCM is fully integrated into the diffusers library, providing access to:
- Native
mpssupport for Apple Silicon. - Performance optimizations including
torch.compile()and flash attention. - Memory-saving strategies such as model offload for low-RAM environments.
- Support for ControlNet and image-to-image workflows.
Combining LoRAs
Using the diffusers and PEFT integration, users can combine LCM LoRAs with regular SDXL LoRAs. This allows specialized style or subject LoRAs (such as CiroN2022/toy_face) to benefit from the 4-step inference speed of the LCM process by setting multiple adapters and their respective weights.
Available Models and Resources
Hugging Face has released several LCM LoRAs and full models:
- LCM LoRAs: Available for SDXL 1.0 base, Stable Diffusion v1.5, and Segmind's SSD-1B (a distilled SDXL model).
- Full Models: Full fine-tuned consistency models derived from SDXL 1.0 base and SSD-1B.
Training and fine-tuning scripts are now available within the diffusers repository for both Stable Diffusion 1.5 and SDXL, allowing the community to perform either full-model distillation or the more accessible LCM LoRA training.