apple-aiml-research/ml-gmpi

What it solves

This project addresses the challenge of making 2D Generative Adversarial Networks (GANs) 3D-aware. It allows for the generation of images that possess 3D structure and depth, enabling the creation of synthetic 3D-like views from a 2D generator.

How it works

GMPI uses Multiplane Images (MPIs), which represent a 3D scene as a collection of semi-transparent layers (planes) with RGB and alpha (transparency) maps. The system can render these planes to produce a 2D image from various camera angles, rendering videos with camera motion, and even extracting a 3D mesh from the generated content.

Who it’s for

Researchers and developers working in computer vision, 3D scene reconstruction, and generative AI, specifically those interested in enhancing 2D GANs with 3D awareness.

Highlights

• 3D-Aware Generation: Transforms 2D GAN outputs into 3D-aware representations.
• Multi-View Rendering: Supports rendering single images, videos with horizontal or vertical camera movement, and depth maps.
• Mesh Extraction: Ability to extract a 3D mesh (.ply file) using Marching Cubes.
• Pretrained Checkpoints: Provides pretrained models for datasets like FFHQ, AFHQCat, and MetFaces.