nicolasvonluetzow/GaussianGPT

What it solves

GaussianGPT addresses the challenge of 3D scene generation by moving away from diffusion or flow-matching models. Instead, it uses a fully autoregressive approach to generate 3D Gaussian primitives, allowing for step-by-step scene construction. This enables capabilities like scene completion, outpainting, and controllable sampling that are more naturally supported by autoregressive transformers than holistic refinement methods.

How it works

The project implements a two-stage pipeline:

1. VQ-VAE: A sparse 3D convolutional autoencoder with vector quantization compresses 3D Gaussian primitives into discrete tokens. This is trained using a re-rendering loss via gsplat.
2. GPT: An autoregressive transformer with 3D rotary positional embeddings models these token streams through next-token prediction.

During inference, the GPT samples tokens which are then decoded by the frozen VQ-VAE back into 3D Gaussians and rendered using standard neural rendering pipelines.

Who it’s for

Researchers and developers working on 3D generative AI, neural rendering, and the creation of synthetic 3D environments.

Highlights

• Autoregressive Generation: Generates 3D scenes sequentially rather than refining them all at once.
• Flexible Control: Supports outpainting, scene completion, and temperature-based sampling.
• Discrete Latent Grid: Uses a sparse 3D CNN to tokenize spatial structure and appearance.
• Scalable Architecture: Leverages the compositional inductive biases of transformers for 3D scene generation.