NVIDIAGameWorks/kaolin

A PyTorch Library for Accelerating 3D Deep Learning Research

What it solves

Kaolin is a PyTorch library designed to accelerate 3D deep learning research by providing a set of GPU-optimized, reusable building blocks. It simplifies the process of working with various 3D representations—such as meshes, point clouds, and Gaussian splats—and provides tools for differentiable rendering and physics simulation, which are often complex to implement from scratch.

How it works

Kaolin integrates several specialized 3D operations into a cohesive PyTorch API. It provides representation-agnostic physics simulation via Simplicits, differentiable rendering engines (like nvdiffrast and DIB-R), and a GPU octree acceleration structure (Structured Point Clouds). It also handles the conversion between different 3D formats (e.g., voxel grids to meshes) and provides utilities for quaternion math and USD I/O.

Who it’s for

It is primarily aimed at researchers and developers working in 3D deep learning, computer vision, and computer graphics, particularly those using PyTorch.

Highlights

  • Representation-Agnostic Physics: Supports simulation of meshes, splats, and point clouds with collisions.
  • 3D Gaussian Splatting: First-class support for GaussianSplatModel, including PLY/USD I/O and densification.
  • Differentiable Rendering: Includes tools for PBR, spherical harmonics, and spherical gaussians lighting.
  • USD Pipeline: Custom schema for importing and exporting meshes, point clouds, and physics materials.
  • Fast Conversions: GPU-accelerated transitions between meshes, voxel grids, point clouds, and gaussians.
  • Structured Point Clouds: GPU octree acceleration for ray tracing and feature grids.

Related

  • Project
    facebookresearch/pytorch3d

    A PyTorch-based library for 3D Computer Vision research providing differentiable rendering and efficient tools for manipulating 3D meshes and point clouds.

  • Project
    neka-nat/cupoch

    A GPU-accelerated library for rapid 3D data processing in robotics, providing high-performance point cloud algorithms, collision avoidance, and path planning.

  • Project
    frgfm/holocron

    A library of high-quality PyTorch implementations of recent deep learning tricks and model architectures for computer vision tasks.

  • Project
    nerfstudio-project/gsplat

    A CUDA-accelerated library for the rasterization of 3D Gaussians, providing a faster and more memory-efficient way to render radiance fields compared to the official implementation.

  • Project
    nnaisense/evotorch

    A scalable evolutionary computation library built on PyTorch that solves non-differentiable optimization problems using GPU acceleration and Ray distribution.