hzxie/CityDreamer

CityDreamer – Generative Model for Unlimited 3‑D Cities

What it is – CityDreamer is a research‑grade, compositional generative model that can synthesize unbounded, photorealistic 3‑D cityscapes. It does this by stitching together three learned components:

1. Unbounded Layout Generator – produces a global city layout (road network, block arrangement) using a VQ‑VAE + autoregressive sampler.
2. Background‑Stuff Generator – fills the layout with terrain, vegetation, sky, and other non‑building elements.
3. Building Instance Generator – creates detailed building geometry and textures for each block.

The system is built on PyTorch (tested with 1.13.1 + CUDA 11.7) and runs on a single high‑memory GPU (e.g., RTX 3090) for inference, while training assumes multi‑GPU setups.

---

Quick start (inference)

1. Install the required PyTorch version, clone the repo, and run pip install -r requirements.txt.
2. Compile the CUDA extensions located in extensions/ (each sub‑folder is installed with pip install .).
3. Run the demo:

   python -m demo.run   # opens a web UI at http://localhost:3186
   

   or use the CLI:

   python -m scripts.inference   # outputs output/rendering.mp4
   

   Adjust --patch_height / --patch_width if your GPU has less than 24 GB VRAM.

---

Training (research use)

1. Datasets – CityDreamer ships two large‑scale datasets:
   • OSM (OpenStreetMap) road‑network data.
   • GoogleEarth aerial imagery with instance‑segmentation masks (generated via SEEM). Both are downloadable via the provided gateway links.
2. Prepare the GoogleEarth annotations (≈1 TB) by running the SEEM segmentation script and then scripts/dataset_generator.py.
3. Train the three modules (each uses the generic run.py entry point with torchrun):
   • VQ‑VAE for layout → 4 GPUs.
   • Sampler for layout → 2 GPUs (loads VQ‑VAE checkpoint).
   • Background‑Stuff Generator → 8 GPUs (set BUILDING_MODE=False).
   • Building Instance Generator → 8 GPUs (set BUILDING_MODE=True). Configuration tweaks are shown in the README under each section.

---

Pre‑trained models & resources

Component	Checkpoint (download)
Layout Generator (Sampler)	LayoutGen.pth
Background‑Stuff Generator	CityDreamer-Bgnd.pth
Building Instance Generator	CityDreamer-Fgnd.pth

A live demo is hosted on Hugging Face Spaces, and a short spotlight video is linked on YouTube.

---

License

The code is released under the NTU S‑Lab License 1.0 (see LICENSE in the repo). Redistribution must follow the terms of that license.

---

Who’s behind it

Developed by Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu at the S‑Lab, Nanyang Technological University. The work was presented at CVPR 2024 and has a follow‑up (CityDreamer4D) accepted to TPAMI.

---

Bottom line – CityDreamer provides a full pipeline (data preparation, training scripts, and inference demo) for researchers interested in large‑scale, procedural generation of realistic 3‑D urban environments.