apple-aiml-research/ml-lito

What it solves

LiTo addresses the challenge of representing 3D objects in a way that captures both their physical geometry and their view-dependent appearance (such as lighting reflections and specular highlights) within a single, compact latent space. Traditional 3D representations often struggle to separate these two elements or fail to reproduce complex optical effects like Fresnel reflections.

How it works

The system uses "Surface Light Field Tokenization" to encode random subsamples of RGB-depth images into a set of latent vectors. By treating RGB-depth images as samples of a surface light field, the model learns a unified 3D latent space that jointly models geometry and appearance. This allows the model to decode latents into various formats, including 3D Gaussians, meshes, and resampled point clouds. Additionally, an image-to-3D generative model (DiT) is trained to produce 3D objects from a single image, ensuring the generated object's appearance is consistent with the lighting and materials of the input image.

Who it’s for

This project is designed for researchers and developers working in 3D computer vision, generative AI, and graphics, specifically those interested in high-fidelity 3D reconstruction and image-to-3D generation.

Highlights

• View-Dependent Effects: Capable of modeling complex lighting, specular highlights, and Fresnel reflections.
• Fast Generation: Achieves image-to-3D generation in approximately 4.7 seconds on an NVIDIA H100.
• Image Alignment: Generated 3D objects are aligned with the input image rather than being placed in an arbitrary coordinate frame.
• Cross-Platform Support: Provides an interactive demo compatible with both Linux (NVIDIA GPU) and macOS (Apple Silicon via MLX).