madebyollin/taehv

Tiny AutoEncoder for Hunyuan Video (and other video models)

🥮 Tiny AutoEncoder for Hunyuan Video (TAEHV)

What it is – TAEHV is a compact auto‑encoder that can encode video frames into latent representations and decode those latents back into video much faster and with far lower GPU memory than the original, full‑size VAEs that ship with models such as Hunyuan Video, MiniMax‑H3, Wan‑Video, CogVideoX, Open‑Sora, LTX‑2, etc.

Why it matters – Full‑size video VAEs typically need 6‑9 GB of GPU memory and several seconds to decode a short clip (e.g., 61 frames of 512×320). TAEHV brings that down to <0.5 GB and ≈0.5 s on an NVIDIA GH200, making real‑time preview, interactive generation, or live‑stream processing feasible.

Supported video models

Model Weight tag you load Example notebook
MiniMax H3 taeh3 –
Hunyuan Video 1.5 taehv1_5 (or taehv1_5_super for higher quality) TAEHV1.5_Encoding_Decoding_Demo.ipynb
Wan 2.1 taew2_1 –
Wan 2.2 5B taew2_2 (or taew2_2_super) TAEW2.2_T2I_Demo.ipynb
Wan 2.2 14B taew2_1 (uses Wan 2.1 VAE) –
Qwen Image taew2_1 –
CogVideoX taecvx TAECVX_T2I_Demo.ipynb
Hunyuan Video 1 taehv TAEHV_T2I_Demo.ipynb
Open‑Sora 1.3 taeos1_3 –
LTX‑2 taeltx_2 TAELTX2_Encoding_Decoding_Demo.ipynb
LTX‑2.3 / 2.5 taeltx2_3 (or taeltx2_3_wide for less blur) –
Mochi 1 & SVD see companion repos TAEM1 and TAESDV

The repository ships the .pth weight files at the root and equivalent .safetensors files in ./safetensors. TAEHV automatically selects the correct architecture from the checkpoint name; if you rename a file you can pass arch_name= manually.

How to use

from taehv import TAEHV

# Load a checkpoint (e.g., Hunyuan‑Video‑1.5)
vae = TAEHV("taehv1_5.pth")

# Encode a video tensor (N, C, T, H, W) -> latent
latents = vae.encode_video(video_tensor)

# Decode back to video
decoded = vae.decode_video(latents)
  • Speed tricks – disable up‑scaling:
vae = TAEHV(decoder_time_upscale=(False, False),
            decoder_space_upscale=(False, False, False))
  • Parallel decode – set parallel=True in decode_video to process all frames at once (needs more memory).
  • Streaming – for live or step‑wise processing use the StreamingTAEHV wrapper, which yields each frame as soon as it is decoded. See the Streaming demo notebook.
  • Diffusers integration – a small wrapper (see TAEW2.1_Diffusers_Encoding_and_Decoding_Demo.ipynb) adapts the differing value ranges ([0,1] vs [-1,1]) and tensor layouts (NTCHW vs NCTHW).

Where it lives

  • ComfyUI – built‑in via the main branch (PR #10884) and through community nodes: ComfyUI‑WanVideoWrapper, ComfyUI‑VideoHelperSuite, ComfyUI‑Bleh.
  • stable‑diffusion.cpp – added via PR #937.
  • SDNext – integrated by vladmandic.
  • Wan‑Video Self‑Forcing demo – used in the official Wan 2.1 demo.

If you add support elsewhere, the author invites a pull request to list it.

Limitations

  • Quality trade‑off – TAEHV is faster and lighter but does not reach the visual fidelity of the original VAEs. The author provides a benchmark (MoViRec) and a side‑by‑side comparison showing the gap.
  • Super‑weights – higher‑quality variants (*_super) exist for some models; they consume a bit more compute and memory.

Getting started resources

  • Profiling notebook – examples/TAEHV_Profiling.ipynb (runtime & memory numbers).
  • Demo notebooks – various *_Demo.ipynb files showing text‑to‑video, encoding/decoding, streaming, and Diffusers usage.
  • Model weights – download from the repo root or the safetensors/ folder.
  • Citation – a ready‑to‑paste BibTeX entry is provided for academic use.

TL;DR

TAEHV is a drop‑in, lightweight VAE replacement for a growing list of Chinese and open‑source video diffusion models. It cuts decoding time from seconds to sub‑second and memory from ~8 GB to <0.5 GB, enabling real‑time preview and interactive video generation, at the cost of a modest quality drop that can be mitigated with the optional “super” checkpoints.

Related

  • Dispatch
    State of Open Video Generation Models in Diffusers

    Hugging Face provides a comprehensive overview of open video generation models and introduces a suite of Diffusers optimizations that can reduce VRAM requirements for models like HunyuanVideo from 60GB to approximately 6.5GB.

  • Project
    apple-aiml-research/ml-fastvlm

    FastVLM is an efficient vision encoding framework that uses a hybrid encoder called FastViTHD to drastically reduce latency and token count for high-resolution images in Vision Language Models.

  • Project
    ModelTC/LightX2V

    LightX2V is an open‑source inference framework for fast image and video generation (T2V, I2V, T2I, I2I, T2AV, etc.). It supports many state‑of‑the‑art models (MiniMax‑H3, Wan, HunyuanVideo, Qwen‑Image, etc.) and provides speed‑up techniques such as 4‑step distilled LoRAs, FP8/NVFP4 quantization, tensor/sequence parallelism, and disaggregated deployment on a wide range of hardware. Benchmarks show up to 3.9× faster inference than competing frameworks on H100 GPUs. The project offers Docker images, pip install, Gradio/ComfyUI front‑ends, extensive docs, and an online demo.

  • Project
    Tencent-Hunyuan/HunyuanVideo-1.5

    A lightweight 8.3B parameter video generation model that enables high-quality text-to-video and image-to-video synthesis on consumer-grade GPUs.

  • Project
    H-EmbodVis/TurboVLA

    TurboVLA is a lightweight Vision-Language-Action model that enables real-time robotic control at 32 Hz using less than 1 GB of VRAM by replacing LLM-centric architectures with a direct vision-language to action mapping.