Robbyant/lingbot-map

What it solves

LingBot-Map addresses the challenge of streaming 3D reconstruction from video sequences. It provides a way to generate dense 3D point clouds and camera trajectories in real-time (feed-forward) without needing the iterative optimization typically required by traditional SLAM or SfM methods, while maintaining stability over very long sequences (up to 25,000 frames).

How it works

The project uses a Geometric Context Transformer, a feed-forward 3D foundation model that unifies coordinate grounding, dense geometric cues, and long-range drift correction. It employs a paged KV cache attention mechanism (via FlashInfer or SDPA) to handle streaming inference efficiently. To manage memory and maintain accuracy over long distances, it uses a keyframe strategy (caching only every N-th frame) and a windowed inference mode that resets the state to prevent pose collapse.

Who it’s for

It is designed for researchers and developers working on 3D computer vision, robotics, and autonomous navigation who need fast, streaming 3D reconstruction of indoor and outdoor environments.

Highlights

• High-Efficiency Streaming: Capable of stable inference at approximately 20 FPS at 518×378 resolution.
• Long-Sequence Stability: Supports extremely long sequences (e.g., 13-minute indoor walkthroughs) through windowed inference and trajectory memory.
• Integrated Sky Masking: Includes an ONNX-based sky segmentation model to filter out sky points for cleaner outdoor reconstructions.
• Flexible Rendering: Provides both an interactive browser-based viewer (via Viser) and an offline batch rendering pipeline for high-quality flythrough videos.