AlayaLab/AlayaWorld

AlayaWorld – Playable Long‑Horizon Video Generation

What it does – AlayaWorld is a research‑grade video‑world model that can generate minutes‑long, 3‑D‑consistent video from a single starting image, a camera trajectory, and a text prompt. It supports real‑time interaction: you can steer the virtual camera with the keyboard and change the prompt while the video is being generated.

Why it matters – Most video‑generation models either produce short clips or lose spatial consistency over time. AlayaWorld tackles three hard problems at once:

• Interaction – a lightweight AdaLN‑based camera controller lets you move the viewpoint on‑the‑fly.
• Consistency – a 3‑D cache plus a compressed frame‑history embedding keep revisited locations recognizable across long roll‑outs.
• Stability – training on drifted histories and an error‑bank mechanism prevents visual artifacts from compounding over minute‑long sequences.
• Runtime – a distilled few‑step DMD student model runs in real time, enabling a playable experience.

Key components

Component	Role	Notable tech
World model core (alaya/)	Config, data handling, memory structures, trainer logic	Autoregressive (AR) + Distilled Motion Diffusion (DMD) pipelines
LTX‑2.3 stack (ltx2/)	Base transformer, VAE, text encoder	DiT‑style transformer, Gemma‑3 text encoder
Depth‑Anything‑3 (DA3)	Provides depth‑aware spatial memory for the AR path	External repo, integrated via third_party/Depth-Anything-3
ViGeo	Spatial‑memory backbone used in the v1.1 version	Pre‑trained checkpoint ViGeo1.1
Reactor (reactor/)	Serves the model as a live, browser‑based stream where you can move the camera and switch prompts	Docker‑based server
Training stages	Stage 0: cache creation, Stage 1: bidirectional pre‑training, Stage 2a: history encoder, Stage 2b: AR fine‑tuning, Stage 3: DMD distillation	Single unified launcher (scripts/finetune/train.sh)

How to use it

1. Install – one Python environment (tested on PyTorch 2.7.1 + CUDA 12.8). Clone the DA3 repo and install it in editable mode.
2. Download weights – the main inference bundle (merged_infer.safetensors) and the stage‑specific checkpoints are hosted on Hugging Face (links in the README).
3. Run inference – four entry points are provided:
   • Case demo (DA3) – real‑time playable stream (inference/run.sh).
   • AR teacher – 30‑step autoregressive generation with full spatial memory.
   • DMD student – 4‑step distilled generation (fast, real‑time).
   • Bidirectional I2V – image‑to‑video generation without a camera trajectory.
4. Play with the browser demo – build the reactor Docker image and launch reactor run; open the UI to steer the camera and swap prompts while the model streams video.
5. Training – follow the stage‑by‑stage scripts in configs/ and the handbook in docs/vigeo/. Partial training data are also available on Hugging Face.

What’s released

• Full inference code for all four paths.
• Pre‑trained weights for the AR teacher, DMD student, and the shared Stage 1 bidirectional transformer.
• Partial training dataset and a complete training handbook.
• An interactive browser demo (reactor).

License – The code and weights are released under the LTX‑2 Community License (non‑commercial, academic use only). Third‑party models (Gemma‑3, Depth‑Anything‑3) must be obtained under their own licenses.

Who’s behind it – Alaya Lab (core leads Kaipeng Zhang & Chuanhao Li) with a team of researchers and engineers.

---

For more details, see the project page, the three arXiv reports, and the Hugging Face model cards linked in the README.