madebyollin/taehv

🥮 Tiny AutoEncoder for Hunyuan Video (TAEHV)

What it is – TAEHV is a compact auto‑encoder that can encode video frames into latent representations and decode those latents back into video much faster and with far lower GPU memory than the original, full‑size VAEs that ship with models such as Hunyuan Video, MiniMax‑H3, Wan‑Video, CogVideoX, Open‑Sora, LTX‑2, etc.

Why it matters – Full‑size video VAEs typically need 6‑9 GB of GPU memory and several seconds to decode a short clip (e.g., 61 frames of 512×320). TAEHV brings that down to <0.5 GB and ≈0.5 s on an NVIDIA GH200, making real‑time preview, interactive generation, or live‑stream processing feasible.

Supported video models

Model	Weight tag you load	Example notebook
MiniMax H3	`taeh3`	–
Hunyuan Video 1.5	`taehv1_5` (or `taehv1_5_super` for higher quality)	`TAEHV1.5_Encoding_Decoding_Demo.ipynb`
Wan 2.1	`taew2_1`	–
Wan 2.2 5B	`taew2_2` (or `taew2_2_super`)	`TAEW2.2_T2I_Demo.ipynb`
Wan 2.2 14B	`taew2_1` (uses Wan 2.1 VAE)	–
Qwen Image	`taew2_1`	–
CogVideoX	`taecvx`	`TAECVX_T2I_Demo.ipynb`
Hunyuan Video 1	`taehv`	`TAEHV_T2I_Demo.ipynb`
Open‑Sora 1.3	`taeos1_3`	–
LTX‑2	`taeltx_2`	`TAELTX2_Encoding_Decoding_Demo.ipynb`
LTX‑2.3 / 2.5	`taeltx2_3` (or `taeltx2_3_wide` for less blur)	–
Mochi 1 & SVD	see companion repos TAEM1 and TAESDV

The repository ships the .pth weight files at the root and equivalent .safetensors files in ./safetensors. TAEHV automatically selects the correct architecture from the checkpoint name; if you rename a file you can pass arch_name= manually.

How to use

from taehv import TAEHV

# Load a checkpoint (e.g., Hunyuan‑Video‑1.5)
vae = TAEHV("taehv1_5.pth")

# Encode a video tensor (N, C, T, H, W) -> latent
latents = vae.encode_video(video_tensor)

# Decode back to video
decoded = vae.decode_video(latents)

Speed tricks – disable up‑scaling:

vae = TAEHV(decoder_time_upscale=(False, False),
            decoder_space_upscale=(False, False, False))

Parallel decode – set parallel=True in decode_video to process all frames at once (needs more memory).
Streaming – for live or step‑wise processing use the StreamingTAEHV wrapper, which yields each frame as soon as it is decoded. See the Streaming demo notebook.
Diffusers integration – a small wrapper (see TAEW2.1_Diffusers_Encoding_and_Decoding_Demo.ipynb) adapts the differing value ranges ([0,1] vs [-1,1]) and tensor layouts (NTCHW vs NCTHW).

Where it lives

ComfyUI – built‑in via the main branch (PR #10884) and through community nodes: ComfyUI‑WanVideoWrapper, ComfyUI‑VideoHelperSuite, ComfyUI‑Bleh.
stable‑diffusion.cpp – added via PR #937.
SDNext – integrated by vladmandic.
Wan‑Video Self‑Forcing demo – used in the official Wan 2.1 demo.

If you add support elsewhere, the author invites a pull request to list it.

Limitations

Quality trade‑off – TAEHV is faster and lighter but does not reach the visual fidelity of the original VAEs. The author provides a benchmark (MoViRec) and a side‑by‑side comparison showing the gap.
Super‑weights – higher‑quality variants (*_super) exist for some models; they consume a bit more compute and memory.

Getting started resources

Profiling notebook – examples/TAEHV_Profiling.ipynb (runtime & memory numbers).
Demo notebooks – various *_Demo.ipynb files showing text‑to‑video, encoding/decoding, streaming, and Diffusers usage.
Model weights – download from the repo root or the safetensors/ folder.
Citation – a ready‑to‑paste BibTeX entry is provided for academic use.

TL;DR

TAEHV is a drop‑in, lightweight VAE replacement for a growing list of Chinese and open‑source video diffusion models. It cuts decoding time from seconds to sub‑second and memory from ~8 GB to <0.5 GB, enabling real‑time preview and interactive video generation, at the cost of a modest quality drop that can be mitigated with the optional “super” checkpoints.