LingBot‑Map: Feed‑forward 3D foundation model for streaming reconstruction

What it is

LingBot‑Map is a research‑grade, feed‑forward neural network that reconstructs a 3‑D scene from a video (or a folder of images) while the video is streaming. It combines three ideas that are usually separate in NeRF‑style pipelines:

1. Geometric‑Context Transformer – a transformer that jointly reasons about camera pose, dense geometry and long‑range drift. It does this through three mechanisms the authors call anchor context, pose‑reference window and trajectory memory.
2. Paged KV‑cache attention (via the FlashInfer library) – lets the model keep a compact memory of past frames so inference stays fast (≈20 fps on 518×378 images) even for tens of thousands of frames.
3. Streaming‑first design – the model runs in a single forward pass; there is no iterative optimisation loop that revisits earlier frames.

The result is a 3‑D point‑cloud reconstruction that can be visualised live in a browser or rendered offline to video.

---

Who might use it

• Robotics / AR developers who need on‑the‑fly scene mapping from a moving camera.
• Researchers comparing streaming‑reconstruction methods against classic optimisation‑based SLAM or NeRF pipelines.
• Content creators who want to turn long walkthrough videos into 3‑D point clouds without a heavyweight post‑process.

---

Quick start (from the README)

1. Create a conda env and install the exact PyTorch version the project recommends (torch 2.8.0 + CUDA 12.8). The version matters because the NVIDIA Kaolin rendering wheels are built for it.
2. Install the package

   pip install -e .
   

3. (Optional) Install FlashInfer – this gives the fast paged KV‑cache. If you skip it the code falls back to PyTorch’s native SDPA attention.
4. Download a checkpoint from the HuggingFace or ModelScope links (e.g. robbyant/lingbot-map).
5. Run the demo on one of the supplied scenes:

   python demo.py --model_path /path/to/lingbot-map.pt \
       --image_folder example/courthouse --mask_sky
   

   A Visor viewer opens at http://localhost:8080 showing a live point‑cloud reconstruction.

---

Main features (as described in the README)

Feature	What it does	How to enable / tune
Geometric Context Transformer	Unifies coordinate grounding, dense geometry cues and drift correction.	Internally handled; you just pick a checkpoint.
Paged KV‑cache (FlashInfer)	Keeps only selected keyframes in memory, allowing >10 k‑frame streams at ~20 fps.	--keyframe_interval N (store every N‑th frame). Use --backend flashinfer for best speed.
Windowed inference	Slides a fixed‑size memory window when the sequence exceeds the training range (~320 frames).	--mode windowed --window_size 128 --overlap_keyframes 8.
Sky masking	Removes sky points using an ONNX sky‑segmentation model, improving outdoor reconstructions.	Install onnxruntime and add --mask_sky (model auto‑downloads).
Camera pose refinement	Iteratively refines pose per frame; default 4 iterations.	Reduce with --camera_num_iterations 1 for speed.
Offline rendering pipeline	Generates a head‑less MP4 of the reconstructed point cloud for very long videos.	Run demo_render/batch_demo.py with a YAML config or explicit flags.
Visualization options	Point size, confidence threshold, down‑sampling, port, etc.	Flags like --point_size, --conf_threshold.

---

Example use‑cases from the repo

• Indoor walkthrough: 25 000 frames (13 min) rendered offline with a sliding‑window setup.
• Dynamic sequence (Droid‑W): Shows the model handling moving objects; sky masking and stride‑2 sampling are demonstrated.
• Loop‑closure trajectory: A loop scene where the model can close the loop without explicit optimisation.

---

Installation quirks to watch out for

• The exact PyTorch 2.8.0 + CUDA 12.8 build is required for the pre‑compiled Kaolin wheels used by the offline renderer. Using a newer PyTorch works for the interactive demo but will need Kaolin built from source.
• FlashInfer is a pure‑Python package that JIT‑compiles CUDA kernels on first use; the first run may take a few seconds.
• Sky‑mask model (skyseg.onnx) is downloaded automatically from HuggingFace; ensure internet access or place the file manually.
• For GPU‑memory‑constrained machines you can:
  • --offload_to_cpu (default) to move per‑frame predictions off‑GPU.
  • Reduce --num_scale_frames (default 8) to shrink the initial activation peak.

---

Where to find more information

• Paper (conference version) – linked in the badge at the top of the README.
• Technical report on arXiv – gives full algorithmic details.
• Project website – contains additional videos and benchmarks.
• Evaluation scripts – under benchmark/ for KITTI, Oxford Spires, etc.

---

TL;DR

LingBot‑Map is a streaming‑first 3‑D reconstruction transformer that runs in real‑time, supports very long video streams via a paged KV‑cache, and ships with demo scripts, pretrained checkpoints, and an offline renderer. It is a genuine research project in the AI‑driven 3‑D reconstruction space.