yehonathanlitman/Lift4D

What it solves

Lift4D enables the reconstruction of complete 4D assets (3D objects that change over time) from single-view "in-the-wild" videos. It addresses the challenge of creating temporally consistent 3D models from videos where the object may be moving or deforming, fusing multiple single-view estimations into a single, cohesive deformable model.

How it works

The pipeline operates in three distinct stages:

1. Segmentation: Uses SAM 3 to segment the target object from the video based on a text prompt.
2. Causal SAM3D: Performs per-frame 3D reconstruction of the object, utilizing causal latent propagation to maintain temporal coherence between frames.
3. Lift4D Training: Fuses the individual per-frame reconstructions into a single canonical shape combined with a learned deformation field, resulting in a consistent 4D asset.

Who it’s for

This tool is designed for researchers and developers working in computer vision, 4D reconstruction, and digital asset creation who need to turn real-world video clips into animatable 3D models.

Highlights

• In-the-wild capability: Works with custom videos and text prompts for object segmentation.
• Temporal coherence: Employs causal latent propagation to ensure 3D reconstructions remain stable across frames.
• Two-step optimization: Separates the training process into geometry optimization and appearance refinement.
• Occlusion handling: Supports occlusion compositing to inpaint missing parts of the object using scene depths.