zju3dv/Diffuman4D

What it solves

Diffuman4D addresses the challenge of high-fidelity, 4D consistent human view synthesis from sparse-view videos. It allows users to render a human performance from any viewpoint (free-viewpoint rendering) with temporal consistency across frames.

How it works

The project utilizes spatio-temporal diffusion models to generate consistent views of humans from a limited number of input cameras. The system can generate 3D image grids (single frame) or 4D image grids (sequences of frames) from sparse inputs. These generated views can then be used to reconstruct high-quality 3D or 4D Gaussian Splatting (3DGS/4DGS) models for immersive rendering.

Who it’s for

Researchers and developers working in computer vision, human-centric 3D/4D generation, and immersive media who need to synthesize new viewpoints of human performances from limited camera setups.

Highlights

• Spatio-Temporal Diffusion: Uses diffusion models to ensure consistency across both space (different views) and time (frames).
• Sparse-to-Dense View Synthesis: Can transform sparse-view videos (e.g., 4 input cameras) into dense-view grids (e.g., 44 cameras).
• Processed DNA-Rendering Dataset: Provides a meticulously re-annotated version of the DNA-Rendering dataset with 1000+ human multi-view video sequences, including foreground masks and skeleton data.
• 4DGS Integration: Supports workflows for reconstructing 4D Gaussian Splatting models from the synthesized views.