guochengqian/Magic123

[ICLR'24] Official PyTorch Implementation of Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors

What it solves

Magic123 generates high-quality 3D objects from a single 2D image. It addresses the challenge of maintaining 3D consistency while adding rich detail and imagination to the generated object, which often fails when using only one type of diffusion prior.

How it works

The project uses a joint prior approach combining 2D and 3D diffusion priors. It employs a coarse-to-fine pipeline consisting of two stages:

  1. Coarse Stage: Uses a NeRF (Neural Radiance Field) to establish the basic geometry and shape.
  2. Fine Stage: Uses DMTet to refine the texture and high-resolution details.

To improve quality, it can optionally use textual inversion to create a special token for the specific object in the image, and uses depth estimation (via MiDaS) to guide the geometry.

Who it’s for

Researchers and developers working in 3D content creation, computer vision, and generative AI who want to transform single images into textured 3D meshes.

Highlights

  • Joint Priors: Combines 2D and 3D diffusion priors to balance 3D consistency and visual detail.
  • Coarse-to-Fine Pipeline: Two-stage process (NeRF then DMTet) for higher quality output.
  • Mesh Export: Supports exporting final objects as .obj files with materials and textures for use in Blender or MeshLab.
  • Customizable Guidance: Allows tuning of the 2D and 3D prior strengths to control the level of detail versus consistency.