UWMILab/UniWM

UniWM – Unified World Models for Visual Navigation

What it is

• A research codebase accompanying an ECCV 2026 paper that proposes a memory‑augmented world‑model for egocentric visual navigation. The model combines visual foresight (predicting future frames) and planning in a single multimodal autoregressive backbone.

Key ideas

• Unified backbone – unlike modular pipelines, the same transformer‑style model (named Anole) both imagines future observations and decides actions, keeping the imagined visual outcome tightly coupled to the chosen action.
• Hierarchical memory – short‑term perceptual cues are stored alongside a longer‑term trajectory context, allowing the model to reason over extended horizons without losing coherence.
• Multimodal input – egocentric image sequences plus per‑step metadata (actions, poses) are fed to the model; the output can be the next action, the next frame, or both.

Dataset

• The authors release a UniWM Dataset on Hugging Face, comprising several indoor navigation splits (go_stanford, recon, sacson, scand) for training/evaluation and an unseen tartandrive split for generalisation tests. Each trajectory folder contains a series of egocentric JPEG frames and a traj_data.pkl with actions and poses.

How to get started

1. Environment – create a Conda env with Python 3.10 and run bash install.sh (installs PyTorch, dependencies, and the Anole code).
2. Data – run bash download_data.sh to pull all splits into data/.
3. Training – a sample torchrun command is provided; you specify the model (anole), the datasets (comma‑separated), image sequence length (e.g., 784 frames), and other flags. Multi‑GPU training is supported.
4. Evaluation – three modes are available:
   • single‑step (predict one step),
   • task‑level (full trajectory evaluation, optional memory‑bank inference),
   • rollout (autoregressive generation of an entire episode).

Related work

• The repo points to the authors’ earlier projects GOViG (LLM‑driven instruction generation from visual observations) and HA‑VLN (a benchmark that mixes discrete and continuous navigation with social‑awareness constraints).

Who might use this

• Researchers building embodied agents that need to imagine future views while planning.
• Developers interested in experimenting with memory‑augmented transformers for navigation tasks.
• Anyone looking for a ready‑to‑run dataset and training script to reproduce the ECCV 2026 results.

License & citation

• MIT‑licensed. If you use the code or dataset, cite the arXiv paper (2510.08713).

---

All details are taken directly from the repository’s README.