nicolasvonluetzow/GaussianGPT

[ECCV'26 Oral] Our method creates 3D Gaussian scenes completely autoregressively, allowing for generation, completion, and outpainting with the same model.

What it solves

GaussianGPT addresses the challenge of 3D scene generation. While most current methods rely on diffusion or flow-matching, this project provides a fully autoregressive alternative that can generate 3D Gaussian scenes step-by-step, enabling capabilities like scene completion, outpainting, and controllable sampling.

How it works

The system uses a two-stage pipeline:

  1. VQ-VAE: A sparse 3D convolutional autoencoder with vector quantization compresses 3D Gaussian primitives into discrete tokens. This stage is supervised by a re-rendering loss using gsplat.
  2. GPT: A causal transformer with 3D rotary positional embeddings models these token streams via next-token prediction.

To generate a scene, the GPT samples tokens which are then decoded by the frozen VQ-VAE back into 3D Gaussians for rendering.

Who it’s for

This project is designed for researchers and developers working in 3D generative modeling, neural rendering, and computer vision, specifically those interested in autoregressive alternatives to diffusion-based 3D generation.

Highlights

  • Autoregressive Generation: Constructs 3D scenes sequentially rather than refining them holistically.
  • Flexible Generation: Supports unconditional generation, prompt-conditioned completion, and spatial outpainting.
  • Scalable Tiling: Capable of tiling multiple chunks to create large-scale 3D scenes.
  • Explicit Representation: Operates on 3D Gaussians, making it compatible with modern neural rendering pipelines.

Related

  • Project
    cvg/resplat

    ReSplat is a feed-forward recurrent model for 3D Gaussian splatting that iteratively refines scene representations using rendering error as feedback.

  • Dispatch
    Introduction to 3D Gaussian Splatting

    3D Gaussian Splatting is a rasterization technique that enables real-time rendering of photorealistic 3D scenes learned from a small set of images.

  • Project
    lucidrains/autoregressive-diffusion-pytorch

    A PyTorch implementation of autoregressive image generation that eliminates the need for vector quantization by using a diffusion-based approach.

  • Dispatch
    Image GPT

    OpenAI's Image GPT (iGPT) demonstrates that a transformer model trained on pixel sequences can generate coherent images and learn competitive unsupervised visual features without domain-specific architectural priors.

  • Project
    caiyuanhao1998/Open-DiffusionGS

    A single-stage image-to-3D generation and reconstruction framework that integrates Gaussian Splatting into a diffusion denoiser for fast, scalable 3D creation.