ABot‑Recon – Streaming 3‑D Reconstruction with a Fixed Local Context

One‑sentence summary: ABot‑Recon builds a dense 3‑D map from a long video stream while only ever looking at the most recent 12 frames, avoiding any heavyweight long‑range memory.

---

What the project does

• Task – Takes a sequence of RGB images (e.g., a dash‑cam video) and outputs:
  1. The camera pose for every frame (a trajectory).
  2. A local point cloud for each frame, expressed in that frame’s coordinate system.
  3. An optional global point cloud obtained by stitching the local clouds together with the estimated poses.
• Key idea – At each time step the model:
  1. Caches the key‑value features of the previous 11 frames.
  2. Predicts a point map for the current frame.
  3. Predicts the relative pose between the current and previous frame.
  4. Composes the relative poses to keep a running global trajectory.
• Because the model never stores a growing state, inference cost stays constant regardless of how long the video is. A lightweight motion‑visual rotation refiner and a pose‑aware loss keep drift low over thousands of frames.
• Optional loop‑closure – If the video revisits a place, a separate module (using DINOv2‑SALAD descriptors) can detect loop candidates, add pose‑graph constraints, and refine the trajectory.

---

How to get it running (quick start)

# 1️⃣ Create a clean conda env (Python ≥3.10)
conda create -n abot-recon python=3.11 -y
conda activate abot-recon

# 2️⃣ Install PyTorch 2.5.1 + CUDA 12.1 and the package itself
pip install torch==2.5.1 torchvision==0.20.1 \
    --index-url https://download.pytorch.org/whl/cu121
pip install -e .

# 3️⃣ (Optional) Speed‑up with FlashInfer & cuRoPE
pip install flashinfer-python
cd abot_recon/modeling/pi3/models/curope && pip install ninja && python setup.py build_ext --inplace

Run the demo (no loop‑closure)

python demo.py \
  --image-dir examples/images \
  --output-dir outputs/demo \
  --attention-backend auto \
  --no-loop-closure

The command reads the ordered images, streams them through the model, and writes:

• camera_poses.npy (trajectory)
• local_points.pt (per‑frame point clouds)
• confidence.pt (per‑point confidence scores)
• optional global point cloud (world_points.pt) if --save-world-points is used.

Using the Python API

from pathlib import Path
from abot_recon import ABotRecon

imgs = sorted(Path('examples/images').glob('*.jpg'))
model = ABotRecon.from_pretrained('acvlab/ABot-Recon', device='cuda')
result = model.infer(imgs)
print(result.camera_poses.shape)   # (N, 4, 4)

The model automatically downloads the checkpoint from Hugging Face; for offline use place the file at checkpoints/abot_recon.safetensors.

---

Main features & why they matter

Feature	What it gives you
Fixed 12‑frame KV cache	Constant memory & compute, even for hour‑long streams.
Streaming‑only inference	No need for a separate SLAM back‑end; you can run on a single GPU in real time (≈24 FPS on an H100 for 504×280 images).
Optional loop‑closure module	Improves accuracy on revisited scenes without changing the core streaming model.
FlashInfer & cuRoPE acceleration	Up to ~2× speed‑up for the attention kernel and rotary‑position encoding.
Pre‑trained checkpoint on Hugging Face / ModelScope	One‑click download, compatible with the provided API and demo.
Visualization scripts	Export PLY point clouds and bird‑eye‑view trajectory images.
Training recipes (branch train)	Ready‑to‑use configs for fine‑tuning on 18 datasets (not yet released at time of writing).

---

Reported performance (from the paper)

Benchmark	Metric	Result	Setting
Oxford Spires (camera pose)	ATE	4.35 m	Streaming only, no loop closure
Oxford Spires (dense recon)	CD	1.37 m	F1 = 91.81 % (τ = 4 m)
KITTI‑02 (throughput)	FPS	24.45	504×280, NVIDIA H100, storage excluded

The authors also evaluate on KITTI, VBR, 7Scenes, TUM‑Dynamic, showing competitive accuracy while keeping a tiny per‑frame footprint (≈6.7 GiB GPU memory).

---

Who might use this?

• Robotics / autonomous‑driving teams that need a lightweight, streaming 3‑D mapper that can run on a single GPU or an edge accelerator (e.g., the AX650N NPU port).
• AR/VR developers wanting to reconstruct a scene on‑the‑fly without storing the whole video history.
• Researchers exploring long‑horizon perception; the repo provides both inference code and (in the train branch) fine‑tuning pipelines.

---

Where to find more info

• Paper: https://arxiv.org/abs/2608.27529
• Project website: https://amap-cvlab.github.io/ABot-Recon-html
• Demo on Hugging Face Spaces: https://huggingface.co/spaces/acvlab/abot-recon-streaming-3d
• Model weights: Hugging Face (acvlab/ABot-Recon) and ModelScope.
• Hardware port: AXERA‑TECH’s AX650N NPU repo (Docker images, pre‑compiled models).

---

License

• Code: Apache‑2.0
• Model weights: separate MODEL_LICENSE.md (see repo).

---

Bottom line

ABot‑Recon is a genuine research‑grade system for long‑horizon streaming 3‑D reconstruction that deliberately avoids any growing internal memory. It delivers real‑time performance, provides a clean Python API, and includes optional loop‑closure for higher accuracy. The repository is actively maintained (news updates up to Oct 2026) and offers both inference and training resources, making it a solid starting point for anyone building perception pipelines that must run continuously on long video streams.