OpenImagingLab/AnyRecon

What it solves

AnyRecon enables arbitrary-view 3D reconstruction from video. It allows users to generate new, consistent views of a scene from a set of capture views, effectively turning a video into a 3D-aware reconstruction.

How it works

The system uses a video diffusion model (specifically Wan2.1 I2V) combined with LoRA weights to generate the reconstructed views. It integrates with a point-cloud rendering pipeline (Pi3) to create condition videos. These condition videos, derived from capture views, provide the geometric structure that guides the diffusion model to produce the final 3D-consistent reconstruction.

Who it’s for

Researchers and developers working in 3D vision, computer graphics, and AI-driven scene reconstruction who have access to high-VRAM GPUs (approximately 45GB for high-resolution processing).

Highlights

• Arbitrary-View Generation: Creates new perspectives of a scene based on input capture views.
• Video Diffusion Integration: Leverages the video diffusion model Wan2.1 for high-quality visual synthesis.
• Point-Cloud Guidance: Uses Pi3 for point-cloud rendering to ensure geometric consistency.
• Custom Trajectories: Supports custom rendering trajectories by allowing users to specify camera poses via 4x4 matrices.