zju3dv/street_gaussians

Street Gaussians – Dynamic Urban‑Scene Modeling with Gaussian Splatting

What it is

• An open‑source implementation of the ECCV 2024 paper Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting.
• The code builds on the 3‑D Gaussian Splatting paradigm and extends it to handle moving objects in large‑scale street‑level datasets (e.g., Waymo Open Dataset).

Key capabilities

• Dynamic scene reconstruction from monocular video or multi‑camera street‑level data.
• Gaussian‑based representation that can be rendered in real time via rasterization.
• Support for LiDAR depth and sky‑mask generation to improve realism.
• Configurable pipelines for training, evaluation, trajectory rendering, and export to the SIBR viewer format.

How to get started

1. Clone the repo

   git clone https://github.com/zju3dv/street_gaussians.git
   

2. Create a conda environment (Python 3.8) and install PyTorch matching your CUDA version.

   conda create -n street-gaussian python=3.8
   conda activate street-gaussian
   pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 \
       --extra-index-url https://download.pytorch.org/whl/cu116
   

3. Install remaining Python dependencies

   pip install -r requirements.txt
   

4. Build the required sub‑modules (diff‑gaussian rasterizer, simple‑knn, Waymo reader)

   pip install ./submodules/diff-gaussian-rasterization
   pip install ./submodules/simple-knn
   pip install ./submodules/simple-waymo-open-dataset-reader
   python script/test_gaussian_rasterization.py   # sanity check
   

Data preparation

• The authors provide example Waymo scenes (download link in the README). For a full experiment you must download the Waymo Open Dataset training/validation splits and the accompanying tracking predictions.
• A conversion script (script/waymo/waymo_converter.py) turns the raw Waymo files into the format expected by the code. Two example commands are given – one for the small demo set and one for the full validation set (the latter also needs a tracker file).
• Optional preprocessing steps:
  • generate_lidar_depth.py creates synthetic LiDAR depth maps.
  • generate_sky_mask.py builds sky masks using GroundingDINO and Segment‑Anything (SAM). Install GroundingDINO separately and download the SAM checkpoint as instructed.

Running the pipeline

• Configuration – All hyper‑parameters live in config.py. Ready‑made YAML files are in configs/experiments_waymo (for the paper’s monocular video setup) and configs/default.yaml (generic sequences).
• Training

  python train.py --config configs/xxxx.yaml
  # helper scripts
  bash script/waymo/train_waymo_expample.sh   # demo scenes
  bash script/waymo/train_waymo_exp.sh        # full experiments
  

• Rendering

  python render.py --config configs/xxxx.yaml mode {evaluate,trajectory}
  # helper scripts
  bash script/waymo/render_waymo_expample.sh
  bash script/waymo/render_waymo_exp.sh
  

• Visualization – Export a single frame to a PLY file compatible with the SIBR viewer:

  python make_ply.py --config configs/xxxx.yaml viewer.frame_id {frame_idx} mode evaluate
  

Project structure highlights

• lib/ – core implementation (Gaussian splatting, optimizer, data loaders).
• submodules/ – third‑party components compiled as Python packages.
• script/waymo/ – data‑conversion and preprocessing utilities.
• configs/ – YAML files that drive training/evaluation.
• images/pipeline.jpg – visual overview of the end‑to‑end workflow.

Citation If you use this code for research, cite the ECCV 2024 paper:

@inproceedings{yan2024street,
    title={Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting},
    author={Yunzhi Yan and Haotong Lin and Chenxu Zhou and Weijie Wang and Haiyang Sun and Kun Zhan and Xianpeng Lang and Xiaowei Zhou and Sida Peng},
    booktitle={ECCV},
    year={2024}
}