zju3dv/street_crafter

What it solves

StreetCrafter addresses the challenge of synthesizing realistic street view videos from novel trajectories. It allows for the generation of high-quality, controllable street scenes that maintain consistency across different viewpoints, which is often difficult with standard video generation tools.

How it works

The system uses a three-step pipeline:

1. Conditioning: It processes LiDAR data, calibrated images, and object tracklets to create colorized point clouds, which serve as pixel-level conditions for the model.
2. Controllable Generation: A video diffusion model is optimized using observed images and reference embeddings, conditioned on the LiDAR renderings to generate controllable street view videos.
3. 3D Distillation: The generated views from the diffusion model are used as supervision signals to optimize a dynamic 3D Gaussian Splatting (3DGS) representation, enabling the rendering of novel trajectories.

Who it’s for

This project is designed for researchers and developers working in computer vision, autonomous driving simulation, and 3D scene reconstruction.

Highlights

• LiDAR-Guided: Uses colorized point clouds to provide precise spatial conditioning for video generation.
• Hybrid Approach: Combines the generative power of video diffusion models with the structural consistency of 3D Gaussian Splatting.
• Novel View Synthesis: Capable of rendering street scenes from entirely new trajectories.
• Dynamic Representation: Supports the distillation of diffusion-generated content into a dynamic 3D representation.