zhouhengamerica/XLens

What it solves

X-Lens addresses the difficulty of estimating depth from a mix of different camera types (heterogeneous rigs). Traditionally, models are trained for either pinhole (perspective) or fisheye (wide-FoV) cameras, requiring separate models or lossy rectification (undistorting fisheye images to look like pinhole images) to work together. X-Lens allows a single model to process any combination of these cameras in one forward pass to recover dense metric depth and a fused 3D point cloud.

How it works

X-Lens uses an alternating frame-to-global cross-view attention backbone. Instead of rectifying images, it represents every pixel as a unit-ray and uses camera-type conditioning (calibration tokens and geometric distortion bias) to handle different lens models.

Key technical components include:

• Per-pixel ray representation: Uses a Look-Up Table (LUT) for fisheye cameras to map pixels directly to viewing directions.
• Ray-angle rotary position encoding (RoPE): Helps the model reason about the geometry of wide-FoV views.
• Learned scale head: Predicts a scale factor to convert relative depth into absolute metric depth.
• Point cloud fusion: Unprojects depth maps from multiple views into a single, consistent world-frame point cloud.

Who it’s for

This project is for researchers and developers working on 3D scene reconstruction, robotics, and autonomous systems that use multi-camera rigs combining standard and wide-angle lenses.

Highlights

• Unified Model: One checkpoint handles pure pinhole, pure fisheye, and mixed heterogeneous setups without retraining.
• Native Fisheye Support: Processes wide-FoV images directly, avoiding the quality loss associated with undistortion.
• Metric Depth Output: Provides absolute distance measurements and confidence maps.
• ONNX Export: Supports baking camera geometry into the graph for high-performance deployment via TensorRT or ONNX Runtime.
• OmniScene Dataset: Includes a photorealistic synthetic dataset of 6-camera omnidirectional rigs for training and evaluation.