EdenLive – Real‑time generative rendering from any visual reference

What it is – EdenLive is a research‑grade, real‑time video generator that takes a reference video (a cheap proxy of a game or simulation) and, guided by a first‑frame image and a text prompt, produces high‑quality, photorealistic frames on‑the‑fly. The upstream program (a game engine, physics simulator, or any code that updates world state) only needs to output a low‑fidelity video; EdenLive handles the visual synthesis.

---

Key capabilities

Feature	Detail
Real‑time inference	>24 fps at 720p on a consumer RTX 5090 (5 B model) and 480p on a 14 B model.
Two inference modes	Bidirectional – all frames denoised jointly; Few‑step (chunk‑AR) – chunked autoregressive sampling (4 steps per chunk) for lower latency.
Model sizes	5 B and 14 B DiT diffusion models, each with a bidirectional and a few‑step variant.
Decoders	LightTAE‑based TAEHV decoder for streaming output; also supports FLUX‑2 first‑frame rendering (non‑commercial).
End‑to‑end pipeline	Includes a skill (EdenGolem) that turns a text prompt into a playable game, a web frontend, and a WebRTC streaming service.
Integration	Ready‑to‑use with the Reactor runtime, NVIDIA FlashDreams, and the LightX2V acceleration library.
Open‑source	Code, inference scripts, real‑time service, and model weights (hosted on Hugging Face).

---

Quick start (high‑level)

1. Clone & set up environment

   git clone https://github.com/VAST-AI-Research/EdenLive.git EdenLive
   cd EdenLive
   conda create -n edenlive python=3.10 -y && conda activate edenlive
   # Install PyTorch for CUDA‑13 (or CUDA‑12.8) and other deps as described in the README.
   

2. Download model weights (≈210 GB total)

   hf download VAST-AI/EdenLive --local-dir ./models/EdenLive
   

3. Run a demo – the repo ships two proxy recordings. For example, bidirectional 5 B at 720p:

   python -m edenlive.scripts.infer \
     --config edenlive/configs/wan22_5b_720p_bidir.yaml
   

   The output video will be written to the outputs/ folder.
4. Start the real‑time service (optional, for interactive games)

   cd realtime
   bash turn_server.sh          # TURN relay for WebRTC
   bash workbench_bridge.sh    # Claude‑based bridge (optional)
   bash workbench_flux.sh      # FLUX first‑frame service (non‑commercial)
   bash launch.sh 5b            # Model service
   

   Then open http://<host>:8089 in a browser to load the web UI.

---

Repository layout (important folders)

• edenlive/ – offline inference code, model wrappers, and utilities.
• realtime/ – streaming service, WebRTC glue, FlashDreams integration, and the EdenGolem skill.
• reactor/ – optional integration with the Reactor runtime for synchronized draft‑and‑generate playback.
• examples/samples/ – two ready‑to‑run proxy recordings with manifests.
• models/EdenLive/ – expected location for downloaded weight files.

---

Model zoo

Model	Size	Mode	Typical resolution
dit-5b	5 B	Few‑step (chunk‑AR)	720p (1280×704)
dit-5b-bidir	5 B	Bidirectional	720p
dit-14b	14 B	Few‑step	480p
dit-14b-bidir	14 B	Bidirectional	480p

All models are hosted on Hugging Face under VAST-AI/EdenLive. The repository also contains the required VAE, CLIP, DINOv3 vision encoder, and FLUX‑2 first‑frame LoRA.

---

Licensing

• Code – Apache 2.0 (including the realtime/ sub‑module).
• Model weights – mixed licenses:
  • DiT, VAE, UMT5, scheduler, decoder: Apache 2.0
  • CLIP: MIT
  • Vision encoder (DINOv3): DINOv3 license
  • FLUX‑2 (first‑frame only): Non‑Commercial license (requires content filtering).

Make sure any commercial use respects the non‑commercial restriction of the FLUX component.

---

Who might use this?

• Game developers who want a cheap proxy engine for physics and AI, while delegating visual fidelity to a generative model.
• Researchers exploring interactive world models, video‑to‑video diffusion, or real‑time text‑guided rendering.
• Artists looking for a live “text‑to‑game” experience where a simple description evolves into a playable scene.

---

Citation – If you publish work using EdenLive, cite the provided BibTeX entry.