LongLive – NVIDIA’s Suite for Long‑Context Video & Robot‑Action Modeling

TL;DR

LongLive is a collection of four related research projects from NVIDIA that push the limits of very long video generation and long‑context robot‑action models. Each sub‑project lives in its own folder and can be cloned independently. The repo provides code, training recipes, inference pipelines, and pretrained checkpoints for:

• Long‑WAM – world‑action models that can watch and act over minutes of video, with real‑robot demos.
• LongLive‑Plug – “once‑for‑all” LoRA‑style distillation that adds new capabilities to existing video models without retraining them.
• LongLive 2.0 – a high‑performance infrastructure (NVFP4 quantisation, sequence‑parallelism, KV‑cache tricks) for training and serving long‑video diffusion models.
• LongLive 1.0 – an interactive, real‑time text‑to‑video system that lets a user steer a video as it streams.

All projects are open‑source under Apache‑2.0 and ship with model weights on HuggingFace.

---

1. What problem does it solve?

Generating or using video that lasts minutes (instead of a few seconds) quickly runs into two bottlene‑cks:

1. Memory – transformer‑style diffusion models keep a KV‑cache that grows linearly with length.
2. Compute – training on long sequences is expensive and inference becomes slow.

LongLive tackles both:

• Streaming visual memory (Long‑WAM) lets a robot keep a rolling representation of what it has seen, enabling policies that act over long horizons.
• NVFP4 quantisation & sequence parallelism (LongLive 2.0) cut memory and speed up training/inference by > 18 %.
• Once‑for‑all distillation (LongLive‑Plug) separates reusable capabilities (LoRA adapters) from the base model, so you can add new behaviours without re‑training the whole diffusion network.
• Interactive generation (LongLive 1.0) streams the diffusion process, allowing a user to change prompts on‑the‑fly.

2. Core components & how they fit together

Component	Goal	Key tech	Typical use‑case
Long‑WAM	Long‑context robot policies & video generation	Streaming IDM/COD, Agentic API, visual memory, robot‑specific adapters	Train/evaluate policies on LIBERO, RoboTwin, RoboCasa; deploy on YAM, Franka, Unitree G1
LongLive‑Plug	Re‑use a trained capability across many video models	LoRA adapters, few‑step / CFG distillation, “once‑for‑all” training	Add a new style, object, or control signal to a base diffusion model without full retraining
LongLive 2.0	Fast, scalable training & inference for long videos	NVFP4 (W4A4) quantisation, KV‑cache quantisation, sequence parallelism, async decoding	Train 5‑B video diffusion models at 45 FPS on a single H100; serve them with low latency
LongLive 1.0	Real‑time interactive text‑to‑video	Attention sink, KV‑re‑cache, streaming long‑tuning	User‑guided video creation where prompts can be edited while the video streams

All four share a common code base (utility scripts, model‑loading helpers, evaluation tools) and can be sparse‑checked‑out individually, so you only pull the part you need.

3. Getting started (quick‑start example)

# Clone only the Long‑WAM sub‑project
git clone --filter=blob:none --sparse --depth 1 \
    https://github.com/NVlabs/LongLive.git && cd LongLive
git sparse-checkout set Long-WAM
cd Long-WAM

# Install dependencies (see Long-WAM/README for exact versions)
pip install -r requirements.txt

# Download a pretrained checkpoint (e.g., LIBERO‑IDM)
python download_checkpoint.py --model Long-WAM-LIBERO-IDM

# Run an inference demo (streaming robot video)
python demo_robot_video.py --policy libe…

Replace Long-WAM with LongLive-Plug, LongLive2.0, or LongLive1.0 to try the other tools.

4. Notable results & benchmarks

Project	Model size	Speed (FPS)	VBench score	Highlight
LongLive‑2.0‑5B‑NVFP4‑2Step	5 B	45.7	83.14	Fastest NVFP4 inference (W4A4)
LongLive‑2.0‑5B‑NVFP4‑4Step	5 B	29.7	84.51	Good trade‑off between speed & quality
LongLive‑1.3B	1.3 B	20.7	84.87	Real‑time interactive generation on a single H100
Long‑WAM (RoboCasa GR1, 4.8 s context)	–	–	–	Demonstrates streaming visual memory over > 10 s of robot video

The repo also lists many downstream projects that have built on LongLive (DreamForge‑World, SANA‑Video, MemFlow, etc.), showing its impact on the community.

5. Who should use this?

• Researchers exploring long‑horizon video diffusion, robot policy learning, or memory‑efficient transformers.
• Engineers building real‑time video‑generation products who need low‑latency inference.
• Robotics teams that want a pretrained world‑action model they can fine‑tune or directly deploy on hardware like Franka or Unitree.
• ML‑ops folks interested in large‑scale parallel training (NVFP4, sequence parallelism).

6. Limitations & open questions

• The most performant configurations (NVFP4, KV‑cache quantisation) currently require NVIDIA H100 / DGX hardware; running on consumer GPUs will be slower.
• Real‑robot deployment examples are limited to the listed platforms (YAM, Franka, Unitree G1). Adapting to other robots will need custom wrappers.
• “Once‑for‑all” distillation works best when the base model and target model share architecture; cross‑architecture transfer is not yet supported.

7. License & citation

The code is released under Apache‑2.0. Each sub‑folder carries its own copy of the license and any third‑party notices. If you use the models or code in research, cite the four arXiv papers listed in the README.

---

Bottom line: LongLive is a genuine, production‑grade research suite that pushes the frontier of long‑context video diffusion and robot‑action modeling, offering both high‑performance infrastructure and ready‑to‑use pretrained models.