pypose/bae

[T-RO 2026] Bundle Adjustment in the Eager Mode

What it solves

bae is a PyTorch-based library designed for high-performance, exact 2nd-order optimization in robotics. It specifically addresses the computational challenges of large-scale sparse optimization problems, such as Bundle Adjustment (BA) and Pose Graph Optimization (PGO), which are essential for 3D reconstruction and spatial awareness.

How it works

The library implements efficient sparse block matrix operations and custom CUDA kernels to accelerate sparse linear algebra. It integrates seamlessly with PyTorch's automatic differentiation framework and provides two primary optimization strategies:

  • Levenberg-Marquardt (LM) Optimizer: A standard algorithm for solving non-linear least squares problems.
  • Schur Complement Optimizer: A method to reduce the linear system size, significantly optimizing memory consumption.

It also supports torch.compile for reducing runtime overhead and latency, and provides a shared API with PyPose for sparse Jacobians.

Who it’s for

This tool is intended for robotics researchers and developers working on SLAM (Simultaneous Localization and Mapping), 3D reconstruction, and spatial AI, particularly those needing GPU-accelerated optimization of camera poses and 3D points.

Highlights

  • CUDA Acceleration: Uses custom kernels for high-performance sparse linear algebra.
  • PyTorch Integration: Works directly with PyTorch's autograd system.
  • Exact 2nd-Order Optimization: Supports precise optimization techniques for BA and PGO.
  • Memory Efficiency: Includes a Schur complement optimizer to handle large-scale problems with lower memory footprints.

Related

  • Project
    pypose/pypose

    A PyTorch-based library that combines deep perceptual models with physics-based optimization for robot learning, providing tools for Lie groups, filtering, and second-order optimization.

  • Project
    pytorch/ao

    A PyTorch-native library for model optimization that provides quantization and sparsity tools to accelerate training and inference while reducing memory usage.

  • Project
    meta-pytorch/botorch

    A PyTorch-based library for Bayesian Optimization that provides a modular framework for composing probabilistic models and acquisition functions.

  • Project
    HomebrewML/HeavyBall

    A high-performance PyTorch optimizer library that uses compiled, composable building blocks to fuse operations into Triton kernels for faster training and lower memory usage.

  • Project
    SimplexLab/TorchJD

    A PyTorch library for training neural networks with multiple losses, providing tools for scalarization and Jacobian descent to resolve optimization conflicts in multi-task learning.