microsoft/LatentSpatialMemory

Latent Spatial Memory for Video World Models

What it solves

This project addresses the inefficiency and inconsistency of video world models that rely on explicit 3D caches. Traditional methods often require repeated RGB rendering and re-encoding of scene content, which is computationally expensive and memory-intensive.

How it works

Latent Spatial Memory stores persistent 3D scene content directly as latent tokens rather than as RGB images. It builds a persistent latent cache from an initial observation and, for each generated video chunk, reads the target-view memory, uses it during the denoising process, and writes updated static scene content back to the cache.

Who it’s for

Researchers and developers working on video world models, 3D scene generation, and efficient spatial consistency in video synthesis.

Highlights

  • Latent Memory: Persistent 3D scene context is maintained directly in latent space.
  • No RGB Detour: Eliminates the need for repeated render-and-reencode cycles.
  • Memory Lifecycle: Implements a full cycle of initialization, reading, denoising, and updating across generated chunks.
  • Efficiency: Achieves significantly faster generation speeds (10.57x) and drastically lower 3D cache memory usage (55x lower).

Related

  • Project
    microsoft/World-R1

    World-R1 is a reinforcement learning framework that improves 3D geometric consistency in text-to-video generation by aligning generated videos with 3D constraints without changing the base model architecture.

  • Project
    ZiYang-xie/WorldGen

    WorldGen is a tool that generates immersive 3D scenes from text prompts or images in seconds, supporting both Gaussian Splatting and mesh outputs for VR, games, and simulations.

  • Project
    Vchitect/Latte

    A Latent Diffusion Transformer for high-quality video generation, supporting both text-to-video and text-to-image tasks.

  • Project
    TencentCloud/TencentDB-Agent-Memory

    TencentDB Agent Memory is an open‑source memory hub for LLM agents. It captures conversations, extracts reusable skills, builds a Wiki from docs, and indexes code into a CodeGraph. A central UI (Memory Hub) lets teams manage assets, set fine‑grained ACLs, and bind specific memory pieces to individual agents. Agents talk to a uniform proxy, so any framework (Claude Code, DeepSeek Harness, OpenClaw, etc.) can use the same memory without code changes. Benchmarks show a 59 % boost in persona retention. The repo provides one‑click Docker deployment, migration tools, and a roadmap for zero‑config cold starts and more integrations.

  • Project
    Robbyant/lingbot-map

    LingBot‑Map is a feed‑forward transformer that reconstructs 3‑D point clouds from streaming video in real‑time, using a paged KV‑cache (FlashInfer) and optional sky‑masking. It provides interactive and offline demo scripts, pretrained checkpoints, and supports very long sequences via windowed inference.