FLUX.3 Image Klein release: unified text‑to‑image, multi‑reference image‑to‑image, and layout‑control model in Diffusers

TL;DR

FLUX.3 Image Klein is a unified image generation model that handles text‑to‑image, multi‑reference image‑to‑image, and bounding‑box layout editing, and it ships in Hugging Face Diffusers with a fast distilled checkpoint (≈2 s for 1024×1024 on an H100) and a base checkpoint for fine‑tuning.

Getting started

Install the latest Diffusers and Transformers (for the Qwen3‑VL text encoder) and load the distilled checkpoint:

pip install -U diffusers transformers

import torch
from diffusers import ModularPipeline

pipe = ModularPipeline.from_pretrained("black-forest-labs/flux-3-image-klein")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

pipe(prompt="a red fox", output="images")[0].save("fox.png")

The base checkpoint (black-forest-labs/flux-3-image-klein-base) runs 50 inference steps with classifier‑free guidance and is the recommended starting point for LoRA fine‑tuning.

What FLUX.3 can do

Text‑to‑image

The model expects a detailed scene description. The distilled checkpoint generates a 1024×1024 image in 16 steps by default. Example prompt and code are provided in the announcement.

Image‑to‑image

Pass one or more reference images via the image argument and describe the desired changes in the prompt. Multiple references are addressed as <ref_image_0>, <ref_image_1>, …

Layout control

Bounding‑box layout editing is expressed as a JSON list of regions appended to the caption. Each region contains an id, a normalized bbox (y‑first, 0‑1000 scale), and a textual desc. The model draws each description inside its box. For edits, src_bbox and tgt_bbox specify where content comes from and where it should be placed.

Model checkpoints and quantization

Checkpoint Description Default inference
black-forest-labs/flux-3-image-klein Time‑step distilled, fast inference 16 steps, no CFG
black-forest-labs/flux-3-image-klein-base Base model for fine‑tuning 50 steps, CFG

Both checkpoints are available as single‑file safetensors. Two quantized variants (fp8r and nvfp4) can be loaded with Flux3Transformer2DModel.from_single_file and BFLQuantizationConfig. The fp8r variant runs on GPUs with compute capability 8.9+ (Ada, Hopper, Blackwell); nvfp4 targets Blackwell GPUs with FlashInfer kernels.

LoRA fine‑tuning

FLUX.3 supports DreamBooth LoRA scripts for both text‑to‑image and image‑to‑image. A sample command fine‑tunes a tarot‑card style LoRA on the public‑domain multimodalart/1920-raider-waite-tarot-public-domain dataset. Recommended memory‑saving flags include --gradient_checkpointing, --cache_latents, and --use_8bit_adam.

Trained adapters are hosted at diffusers-modular/flux3-tarot-lora. An image‑to‑image LoRA script (train_dreambooth_lora_flux3_klein_img2img.py) trains on (reference, target, instruction) triples, using <ref_image_0> placeholders in the prompt.

Resource usage and performance

Memory footprint

Variant Peak after load (GiB) Peak after inference (GiB)
nvfp4 (RTX 6000) 13.4 15.3
fp8r (H100) 14.35 16.27
bf16 (H100) 18.80 20.72

Quantized checkpoints reduce memory; the text encoder can be run once and freed, and automatic CPU offloading is supported via ComponentsManager.

Speed (single NVIDIA H100 80 GB, no offloading)

Workflow Output size Time Peak memory
Text‑to‑image 1024×1024 1.93 s 23.8 GiB
Image‑to‑image (one reference) 1024×1024 3.87 s 23.8 GiB
Layout editing (knight example) 1360×768 4.10 s 23.8 GiB

A reference image roughly doubles latency because its tokens are concatenated to the text sequence (≈4 k additional tokens per 1024×1024 reference).

Architecture overview

  • Token types – text, noisy image latents, and optional reference latents.
  • Pixel‑to‑token conversion – A VAE compresses 1024×1024 pixels to a 64×64 latent grid (4 096 tokens), each covering a 16×16 pixel patch.
  • Text encoding – Qwen3‑VL provides hidden states from eight of its 36 layers; each token becomes a 20 480‑dim vector, projected to the transformer width of 3072.
  • Two‑stage transformer – Stage 1 processes each modality with its own 5 blocks; Stage 2 merges all streams into 28 joint blocks, enabling cross‑modal attention.
  • Modulation – Timestep embeddings modulate tokens; image tokens use the current timestep, while text and reference tokens are always modulated with timestep 0 (treated as clean context).
  • Attention – Rotary position embeddings span four axes (timestep, height, width, sequence length), giving precise spatial grounding for reference‑based editing.
  • Denoising loop – A flow‑matching scheduler predicts velocity vectors at each step; the distilled checkpoint uses larger jumps (16‑step flow map).

Modular Diffusers integration

FLUX.3 is exposed as a ModularPipeline. Users can:

  • List available workflows (text2image, image_conditioned).
  • Remove the text encoder to encode prompts once and generate multiple images without re‑encoding.
  • Insert custom blocks (e.g., a preview step that records intermediate denoised latents) by extending ModularPipelineBlocks.

Try it yourself

Build, share, and iterate on your own FLUX.3 pipelines and LoRAs on the Hub!

Sources