Segmind SD-Small and SD-Tiny Knowledge Distillation Release

TL;DR

Segmind released the training code and pretrained checkpoints for SD‑Small and SD‑Tiny, diffusion models compressed via block‑removal knowledge distillation; they retain image fidelity comparable to the base model while using 35%–55% fewer parameters and achieving up to 100% faster inference.

Knowledge Distillation Methodology

The released models are trained using the Block‑Removal Knowledge Distillation technique described in On Architectural Compression of Text‑to‑Image Diffusion Models (Shinkook et al.).

  • Teacher model: Realistic‑Vision 4.0, a high‑quality Stable Diffusion checkpoint.
  • Student architectures: UNet variants with removed layers, yielding 35% (SD‑Small) and 55% (SD‑Tiny) fewer parameters.
  • Loss composition:
    1. Standard diffusion loss between latent representations of the target and generated images.
    2. Latent‑level loss aligning student‑generated latents with teacher‑generated latents.
    3. Feature‑level loss matching the outputs of each UNet block between teacher and student (the most critical component).
  • Training data: LAION Art Aesthetic dataset filtered to image scores > 7.5.
  • Training schedule: 1 M images for 100 k steps (SD‑Small) and 125 k steps (SD‑Tiny).

The full distillation pipeline is available in the [segmind/distill‑sd](https://github.com/segmind/distill-sd) repository, and the pretrained checkpoints are hosted on Hugging Face under the segmind namespace.

Model Usage with 🤗 Diffusers

Both models can be loaded directly with the DiffusionPipeline from the 🤗 Diffusers library:

from diffusers import DiffusionPipeline
import torch

pipeline = DiffusionPipeline.from_pretrained(
    "segmind/small-sd", torch_dtype=torch.float16
)
prompt = "Portrait of a pretty girl"
negative_prompt = (
    "(deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, "
    "cartoon, drawing, anime:1.4), text, close up, cropped, out of frame, "
    "worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, "
    "mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn "
    "face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, "
    "extra limbs, cloned face, disfigured, gross proportions, malformed limbs, "
    "missing arms, missing legs, extra arms, extra legs, fused fingers, too many "
    "fingers, long neck"
)
image = pipeline(prompt, negative_prompt=negative_prompt).images[0]
image.save("my_image.png")

The same API works for segmind/tiny-sd by swapping the model identifier.

Inference Speed Gains

Benchmarking on identical hardware shows up to 2× lower latency for the distilled models compared with the original base checkpoint. The inference script used for these measurements is included in the repository (inference.py).

Known Limitations

  • The models are early‑stage releases; visual quality may not yet match production‑grade diffusion models.
  • They are not optimized for general‑purpose generation and may struggle with complex compositional prompts or multiple concepts.
  • Best practice is to fine‑tune or apply LoRA on domain‑specific data to achieve higher fidelity.

Fine‑Tuning SD‑Tiny on a Portrait Dataset

Segmind demonstrated fine‑tuning of SD‑Tiny on a 7 k image portrait set generated with Realistic‑Vision 4.0. Training hyper‑parameters:

  • Steps: 131 000
  • Learning rate: 1e‑4
  • Batch size: 32 (gradient accumulation = 4)
  • Image resolution: 768 px
  • Mixed‑precision: fp16

The resulting samples approach the quality of the original teacher model while retaining the 55% parameter reduction.

LoRA Training on Distilled Models

Applying Low‑Rank Adaptation (LoRA) to SD‑Tiny yields faster LoRA convergence because of the reduced model size. Example LoRA checkpoints trained on abstract concepts are provided in the repo (lora_training.py).

Community Invitation

Segmind encourages developers to contribute to the project, report issues, and share fine‑tuned checkpoints. Communication channels include a Discord server and the GitHub repository, where stars and pull requests are welcomed.

Sources