IamCreateAI/NeoVerse

What it solves

NeoVerse addresses the challenge of creating 4D world models from standard, single-camera (monocular) videos. It allows users to take a regular video and reconstruct the 3D scene and camera motion, enabling the generation of new videos from entirely different camera trajectories (novel-trajectory video generation) and supporting various 4D reconstruction tasks.

How it works

NeoVerse combines two primary components into a pipeline:

1. Reconstructor: It recovers the 3D structure of a scene using Gaussian Splats and estimates camera poses from a monocular video. It is compatible with multiple reconstructors, including a WorldMirror-based version and Depth Anything 3.
2. Video Diffusion Model: Using a WAN 2.1 backbone accelerated by a 4-step distilled LoRA, the system generates high-quality video frames conditioned on the reconstructed 3D scene and a target camera trajectory.

Users can define trajectories using predefined motions (like panning, tilting, or orbiting) or via custom JSON files for precise keyframe control. The system can also handle static scenes (single images) or general scenes with moving objects.

Who it’s for

This tool is designed for researchers and developers in computer vision and video generation who need to synthesize novel views of a scene from a single video, stabilize shaky footage, or expand a single-view video into a multi-view sequence.

Highlights

• Novel Trajectory Generation: Create new camera paths (e.g., orbit, push-in, boom-up) for an existing scene.
• Fast Inference: Completes the pipeline in under 30 seconds on a single A800 GPU using distilled LoRA.
• Plug-and-Play Reconstructors: Supports various 3D reconstructors to estimate scene geometry.
• Flexible Control: Offers both simple predefined camera motions and complex, keyframe-level JSON trajectory control.
• Downstream Applications: Includes tools for camera stabilization, video editing, and single-to-multi-view expansion.