Kandinsky 6.0 – Text‑to‑Audio‑Video Diffusion Models

What it is

Kandinsky 6.0 is a pair of open‑source diffusion models that generate short (5 s) videos with synchronized 44 kHz audio from text or an image prompt. Two sizes are provided:

• Lite – 3 B parameters, fast and runnable on high‑end consumer GPUs.
• Pro – 29 B parameters, higher quality and capable of full‑HD output. A separate super‑resolution plug‑in can upscale the video to 1920 × 1080.

Why it matters

Generating coherent video and matching audio in a single model is still rare. Kandinsky 6.0 tackles the “lip‑sync” problem and supports two generation modes:

• T2AV – text → audio + video.
• TI2AV – image → audio + video (useful for animating a still picture). The models are positioned as a “foundation” diffusion model, meaning they can be fine‑tuned or used as a base for downstream creative tools.

How to get started

1. Clone the repo and install the helper tools (uv for isolated Python environments, just for task automation).
2. Run just setup – this creates a dedicated environment and pulls required dependencies.
3. Download a model checkpoint (just download pro-distill for the distilled 5‑second Pro version).
4. Generate a clip with a single command, e.g.:

   just generate 'A cat is lying on a mat, turns toward the camera, opens its mouth and says <S>Hello<E>'
   

   The special <S>/<E> tokens mark the spoken segment.

The repo ships separate quick‑start guides for different runtimes (plain pipeline, vLLM‑Omni, SGLang, FastVideo) and a justfile that lists all reproducible recipes.

Performance snapshot (5‑second clip, warm‑up excluded)

GPU	Lite SD (s)	Lite HD	Lite Full HD	Pro SD	Pro HD	Pro Full HD
RTX 4090	437	422	578	936	1194	1247
RTX 5060 Ti	1310	1328	1774	3080	2716	3530
RTX 5080	577	579	770	1336	1189	1546
RTX 5090	309	296	406	754	638	854
RTX PRO 6000	242	274	387	621	561	765
A100 80 GB	532	472	664	972	791	1106
H100 80 GB	239	203	284	356	292	402

Numbers are generation time in seconds; weight loading and MP4 encoding are not counted.

Ecosystem integrations

• 🤗 Diffusers – ready‑to‑use pipeline (Kandinsky6 class).
• ComfyUI – node package for visual workflow editors.
• vLLM‑Omni – high‑throughput inference server support.
• SGLang – integration for low‑latency serving.
• FastVideo – optimized inference on the FastVideo framework.
• Hugging Face Space – live demo of the distilled Pro model (5 s generation).

Where to find more

• Paper: https://huggingface.co/papers/2610.05608
• Model cards & checkpoints: https://kandinskylab.ai/models/video/
• Super‑resolution plug‑in: https://github.com/kandinskylab/kandinsky-6-sr
• Detailed docs live in the docs/ folder (quick‑start, justfile, runtime‑specific guides).

---

All information above is taken directly from the repository’s README.