georghess/neurad-studio

neurad‑studio – Neural Rendering for Autonomous Driving

What it is – An open‑source codebase that implements two state‑of‑the‑art neural‑rendering models for self‑driving perception:

NeuRAD – a NeRF‑based method that renders camera images and lidar point clouds from dynamic driving scenes.
SplatAD – a 3‑D Gaussian‑splatting (3DGS) method that achieves real‑time rendering of both lidar and camera data.

It builds on the Nerfstudio framework, adding data parsers, datamanagers, and viewer extensions needed for autonomous‑driving datasets (nuScenes, Waymo, PandaSet, etc.). The repo is meant for researchers who want to train, evaluate, or extend neural‑rendering models on AD data.

Quick‑start (single command flow)

Create a conda env (Python ≥ 3.10) and install CUDA‑11.8, PyTorch 2.0.1, and tiny‑cuda‑nn as described in the README.

Clone & install the package:

git clone https://github.com/georghess/neurad-studio.git
cd neurad-studio
pip install -e .

Download a dataset – e.g. PandaSet from the provided HuggingFace link and unpack it under data/pandaset.
Train a model (NeuRAD example):
```
python nerfstudio/scripts/train.py neurad pandaset-data
```
Replace neurad with splatad to train the 3DGS model.
Watch progress in the built‑in web viewer (default port 7007) or via TensorBoard/WandB.
Render or visualise the trained model with the supplied render.py or run_viewer.py scripts.

Main Features

Feature	NeuRAD	SplatAD
Core rendering	NeRF‑based, high‑quality view synthesis	3D Gaussian splatting, real‑time speed
Sensor support	Camera + lidar (intensity + timestamp)	Camera + lidar (same)
Dynamic objects	Rolling‑shutter ray generation, actor trajectories	Dynamic‑object‑aware 3DGS
Dataset parsers	nuScenes, ZOD, Argoverse 2, PandaSet, KITTIMOT, Waymo v2, py123d	Same parsers reused
Viewer extensions	Lidar point cloud overlay, actor manipulation	Same + faster rendering
Plug‑in models	UniSim (via external repo)	–
Training utilities	TensorBoard, Weights & Biases, Comet, built‑in viewer	Same

Extensibility

Add a new dataset – create a dataparsers/<mydataset>.py that inherits from ADDataParser and implement camera/lidar extraction methods.
Add a new method – follow Nerfstudio’s plugin guide; a new model can be invoked with ns-train <model_name> <data_config>.
Custom kernels – SplatAD relies on a fork of gsplat that handles rolling‑shutter and lidar rendering; the repo provides a Dockerfile and an Apptainer recipe for reproducible environments.

Who might use it?

Research labs exploring neural scene representations for perception, simulation, or sensor‑fusion in autonomous vehicles.
Engineers needing a differentiable simulator to generate synthetic camera/lidar data for downstream tasks (e.g., detection, tracking).
Students learning about NeRF, 3DGS, and their adaptation to dynamic, multi‑sensor driving scenarios.

Citation

If you use the code, cite the two CVPR papers linked in the README:

@inproceedings{tonderski2024neurad,
  title={{NeuRAD}: Neural rendering for autonomous driving},
  author={Tonderski, Adam and Lindström, Carl and Hess, Georg and …},
  booktitle={CVPR 2024},
  year={2024}
}

@inproceedings{hess2024splatad,
  title={{SplatAD}: Real‑Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving},
  author={Hess, Georg and Lindström, Carl and …},
  booktitle={CVPR 2025},
  year={2025}
}

License & Community

The project is released under the same permissive license as Nerfstudio (MIT). Contributions are encouraged; the README lists the core contributors and points to the Nerfstudio contributors for broader community support.