TencentARC/WorldCrafter

[Arxiv 2026] WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

🌍 WorldCrafter – Consistent Video World Model with Implicit 3‑D‑aware Memory

What it is – WorldCrafter is a research‑grade model that can turn a single image or a text prompt into a controllable video scene. The model stores an implicit 3‑D‑aware memory of the scene, so you can move a virtual camera around and keep the generated video consistent over long camera trajectories.

Key capabilities

  • Image‑to‑video (I2V) – give an input picture and a camera path, and the model renders a video that respects the original viewpoint while smoothly revealing new angles.
  • Text‑to‑video (T2V) – start from a textual description and a camera trajectory to synthesize a video from scratch.
  • Camera‑queryable memory – the internal representation is view‑agnostic, enabling consistent appearance and geometry when the camera moves.
  • Two model flavours
    • WorldCrafter‑Base – the full‑size model for highest quality.
    • WorldCrafter‑Fast – a distilled version that runs faster with a small quality trade‑off.
  • Automatic prompt generation – optional integration with Qwen‑3‑VL‑4B‑Instruct can auto‑create first‑person or third‑person scene descriptions from the input image.
  • Open‑source demo & serving stack – a ready‑to‑run interactive web demo (keyboard‑controlled camera) and the code needed to host the model yourself.

📦 Quick start (setup)

  1. Clone the repo
    git clone https://github.com/TencentARC/WorldCrafter.git
cd WorldCrafter
  2. Create a Python 3.11 environment – the README recommends either:
    • uv (the lightweight package manager) – installs a frozen PyTorch 2.10 / CUDA 12.8 stack and ffmpeg.
    • conda + pip – manually install PyTorch 2.10 with the matching CUDA build, then pip install -e "[demo,xformers]" flash-attn-3==3.0.0.
  3. Download model weights from Hugging Face:
    hf download TencentARC/WorldCrafter-Fast --local-dir weights/WorldCrafter-Fast
# optional – also get the Base model
hf download TencentARC/WorldCrafter-Base --local-dir weights/WorldCrafter-Base
# optional – Qwen3‑VL‑4B‑Instruct for auto‑prompt generation
hf download Qwen/Qwen3-VL-4B-Instruct --local-dir weights/Qwen3-VL-4B-Instruct
    Keep both the Fast and Base folders if you plan to switch between them.

🚀 Running inference

The entry point is inference.py. Two modes are supported:

1. Image‑to‑video (I2V)

# using the Base model
python inference.py --model-type base --mode i2v \
  --image-path test/I2V/00_cat_robot_vacuum/image.png \
  --prompt test/I2V/00_cat_robot_vacuum/prompt.txt \
  --camera-path test/I2V/00_cat_robot_vacuum/camera.npy

# using the Fast (distilled) model
python inference.py --model-type fast --mode i2v \
  --image-path test/I2V/03_waterfall/image.png \
  --prompt test/I2V/03_waterfall/prompt.txt \
  --camera-path test/I2V/03_waterfall/camera.npy

You can replace --prompt with auto-first-person or auto-third-person to let the Qwen‑VL model write a description automatically.

2. Text‑to‑video (T2V)

python inference.py --model-type base --mode t2v \
  --prompt test/T2V/00_red_balloon/prompt.txt \
  --camera-path test/T2V/00_red_balloon/camera.npy

Add --enable-compile if you want to trigger Torch‑2 compilation for a (once‑only) speed boost.

🎮 Interactive demo

A browser‑based demo ships with the repo. After activating your environment:

python -m demo --model-path weights/WorldCrafter-Fast

Open http://localhost:8080. The demo uses the Fast model by default, enables Torch‑2 compilation, and supports multi‑GPU (--devices 0,1). Keyboard shortcuts let you pan, tilt, and zoom the virtual camera in real time.

📚 Related work & citations

WorldCrafter builds on a line of neural world‑model research (Helios, LagerNVS, DreamX‑World, etc.). If you use the code or models in research, cite:

@article{yu2026worldcrafter,
  title={WorldCrafter: Consistent Video World Model with Implicit {3D}-aware Memory},
  author={Yu, Wangbo and Liu, Kunhao and Hu, Wenbo and Yuan, Shenghai and Feng, Chaoran and Zhou, Haiyang and Huang, Yukun and Wang, Yiran and Zhao, Wang and Luo, Yingmin and Shan, Ying},
  journal={arXiv preprint arXiv:2609.24984},
  year={2026}
}

📄 License

The repository is released under the license found in LICENSE.txt. It also contains third‑party attributions for components such as Flash‑Attention and XFormers.

Bottom line – WorldCrafter gives researchers and developers a ready‑to‑run pipeline for generating consistent, camera‑controllable videos from a single image or a text prompt, with both a high‑quality base model and a faster distilled variant, plus an open‑source interactive demo for quick experimentation.

Related

  • Project
    Robbyant/lingbot-world-v2

    LingBot‑World‑V2 is a research‑grade video‑generation model that creates infinite, interactive worlds from text prompts. It offers fast (real‑time) and high‑quality variants (14 B and 1.3 B), supports diverse actions, and uses a pilot‑director agent architecture. The repo provides inference code, model download links, and a quick‑start guide; deployment code is not released.

  • Project
    haoyi-duan/WorldScore

    WorldScore is an open‑source benchmark for evaluating world‑generation models (3‑D, 4‑D, video). It provides a dataset, generation adapters, a suite of spatial‑temporal metrics, and a public leaderboard. Users register their model via a small YAML config and a Python class, generate videos, run the evaluation script, and submit the resulting JSON score.

  • Project
    AlayaLab/WorldSculpt

    WorldSculpt is a research codebase that converts multi‑view RGB video (with instance masks and 3D boxes) into a single compositional mesh for scenes containing hundreds of objects. It fine‑tunes the Pixal3D single‑object prior, provides a new UE‑MeshyScene benchmark, releases checkpoints and data on Hugging Face, and includes a community ComfyUI integration.

  • Project
    microsoft/TimeCraft

    TimeCraft is a Microsoft‑Research diffusion‑based framework for generating synthetic time‑series data. It supports cross‑domain few‑shot adaptation via a prototype dictionary, natural‑language control, and target‑aware generation that optimises synthetic samples for downstream tasks. Extensions add causal constraints, online augmentation for foundation‑model pre‑training, and continuous‑time generation from irregular observations.

  • Project
    hao-ai-lab/FastVideo

    FastVideo is an open‑source framework for training, fine‑tuning, and real‑time inference of video diffusion models. It offers sparse‑attention distillation, scalable multi‑GPU training, and cross‑platform support (NVIDIA GPUs, Apple Silicon via MLX, ARM DGX Spark). The library includes a CLI, Python API, and the Dreamverse app for live video editing. Install via `uv` and start generating videos with a few lines of Python.