nv-tlabs/ppisp

Physically-Plausible Image Signal Processing (PPISP) for Radiance Field Reconstruction

What it solves

PPISP addresses the problem of photometric variations in real-world multi-camera video captures used for radiance field reconstruction. It corrects for inconsistencies such as brightness changes due to auto-exposure, radial light falloff (vignetting) from lenses, white balance drift, and nonlinear tone mapping (Camera Response Function) applied by camera hardware.

How it works

It functions as a learned post-processing module that can be integrated into radiance field training pipelines. It models these photometric variations as differentiable CUDA kernels. The system includes a controller that predicts per-frame exposure and color corrections directly from rendered images, allowing for consistent novel view synthesis without needing per-frame parameters during inference.

Who it’s for

Researchers and developers working with radiance fields (such as NeRF or Gaussian Splatting) who need to handle inconsistent lighting and camera settings across multiple video sources.

Highlights

  • Comprehensive Photometric Modeling: Handles exposure, vignetting, color correction, and Camera Response Functions (CRF).
  • Differentiable CUDA Kernels: Implemented for high efficiency within training pipelines.
  • Controller Distillation: Includes a mechanism to predict corrections at inference time, removing the dependency on per-frame parameters.
  • Analysis Tools: Provides utilities to export PDF reports visualizing the learned camera parameters.

Related

  • Project
    Dai-Wenxun/MotionLCM

    MotionLCM is a research codebase for real‑time, text‑conditioned human motion generation. It uses a latent consistency model for one‑step inference and a Motion ControlNet to incorporate explicit joint trajectories, supporting motion reconstruction, text‑to‑motion, and controllable motion synthesis. The README provides conda setup, data preparation (HumanML3D), pretrained model download, demo commands, rendering with SMPL/Blender, and full training instructions via YAML configs.

  • Project
    jasongzy/Make-It-Animatable

    An efficient framework that automates the creation of animation-ready 3D characters by predicting joint positions and skinning weights from 3D meshes.

  • Project
    asanchezyali/talking-avatar-with-ai

    A demo that combines OpenAI GPT, Whisper, Eleven Labs TTS, and Rhubarb Lip‑Sync to create a talking 3‑D avatar with facial expressions and body animations. Install dependencies, set API keys, run `yarn dev`, and interact via text or voice.

  • Project
    apple-aiml-research/ml-lito

    LiTo is a 3D latent representation system that jointly models object geometry and view-dependent appearance, enabling high-quality image-to-3D generation with realistic lighting effects.