MIV-XJTU/JanusVLN

[ICLR2026] Official implementation for "JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation"

What it solves

JanusVLN addresses the challenge of Vision-Language Navigation (VLN), where an agent must navigate a 3D environment based on natural language instructions. It specifically aims to move beyond 2D semantics-dominant approaches by better integrating semantic understanding with 3D spatial cognition to improve navigation accuracy.

How it works

Inspired by the human brain's separation of semantic and spatial processing, JanusVLN utilizes a dual implicit memory system. This system consists of two complementary, fixed-size, compact neural memories that decouple semantic understanding from spatiality, allowing the agent to synergize both types of information for more effective movement in 3D spaces.

Who it’s for

This project is designed for researchers and developers working on spatial embodied AI, robotics, and vision-language navigation agents.

Highlights

  • Dual Implicit Memory: The first VLN framework to use two separate neural memories for semantics and spatiality.
  • 3D Spatial-Semantic Synergy: Shifts the focus from simple 2D semantic recognition to integrated 3D spatial awareness.
  • Comprehensive Training Pipeline: Supports base training, DAgger data collection, and extra training for performance enhancement.
  • Multi-Dataset Support: Compatible with R2R, RxR, and ScaleVLN datasets.

Related

  • Project
    BAAI-DCAI/SpatialBot

    SpatialBot is a Vision-Language-Action model that enhances spatial understanding and depth perception in VLMs, enabling precise spatial reasoning and robot manipulation.

  • Project
    IBM/LNN

    LNN is a Python library for neuro‑symbolic AI that represents logical formulas as weighted neural networks, enabling interpretable, differentiable reasoning and learning even with inconsistent or incomplete knowledge.

  • Project
    bytedance/Sa2VA

    Sa2VA is a ByteDance research codebase that fuses SAM‑2 segmentation with multimodal LLMs (InternVL, Qwen‑VL) to provide pixel‑level grounded understanding for images and videos. It includes the core Sa2VA model, the VRT reasoning benchmark, the SAMTok mask‑token interface, and extensions like SaSaSa2VA. The repo uses the `uv` package manager for reproducible environments, provides setup scripts, and links to papers, model zoo, and datasets.

  • Project
    Wakals/CoVT

    CoVT is a framework that enables Vision-Language Models to reason using continuous visual tokens, improving their spatial reasoning and geometric awareness by grounding semantic thoughts in perceptual cues.

  • Project
    facebookresearch/VLM3

    VLM³ is a framework that enables standard Vision Language Models to perform 3D vision tasks like depth estimation and camera pose estimation using only text-based outputs and data scaling.