luciddreamer-cvlab/LucidDreamer

Official code for the paper "LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes".

What it solves

LucidDreamer enables the generation of 3D scenes using Gaussian Splatting from a single image and a text prompt. It addresses the challenge of creating high-quality 3D environments across various domains without requiring domain-specific training data.

How it works

The system uses an inpainting process guided by text prompts and an input image to "dream" the scene. It leverages models like Stable Diffusion (SD 1.5) and ZoeDepth for depth estimation, and employs a Gaussian splatting rasterizer to convert these generated views into a 3D scene. Users can specify camera paths for both the generation and rendering phases to control the perspective of the resulting 3D output.

Who it’s for

This tool is designed for researchers and developers in computer vision and 3D graphics who want to generate immersive 3D scenes from 2D inputs, as well as artists using AI-driven 3D content creation tools.

Highlights

  • Domain-free generation: Capable of creating 3D scenes across different environments without specific training.
  • Flexible input: Works with a combination of a single image and text prompts.
  • Customizable camera paths: Supports various camera movements (e.g., rotate360, lookaround) for generation and rendering.
  • Integration with SD: Supports custom Stable Diffusion checkpoints for refined inpainting results.

Related

  • Project
    ZiYang-xie/WorldGen

    WorldGen is a tool that generates immersive 3D scenes from text prompts or images in seconds, supporting both Gaussian Splatting and mesh outputs for VR, games, and simulations.

  • Project
    WHU-USI3DV/VistaDream

    VistaDream is a training-free framework that reconstructs high-quality 3D scenes from a single-view image by ensuring consistency across generated novel views.

  • Dispatch
    Introduction to 3D Gaussian Splatting

    3D Gaussian Splatting is a rasterization technique that enables real-time rendering of photorealistic 3D scenes learned from a small set of images.

  • Project
    WebODM/OpenSplat

    A fast, portable C++ implementation of 3D Gaussian Splatting that converts camera poses and sparse points into renderable 3D scene files.

  • Project
    zju3dv/InfiniSplat

    InfiniSplat is a framework for large-baseline monocular view synthesis that reconstructs 3D Gaussian Splatting scenes from a single RGB image or an RGB-depth pair.