microsoft/LatentSpatialMemory

What it solves

This project addresses the inefficiency and inconsistency of video world models that rely on explicit 3D caches. Traditional methods often require repeated RGB rendering and re-encoding of scene content, which is computationally expensive and memory-intensive.

How it works

Latent Spatial Memory stores persistent 3D scene content directly as latent tokens rather than as RGB images. It builds a persistent latent cache from an initial observation and, for each generated video chunk, reads the target-view memory, uses it during the denoising process, and writes updated static scene content back to the cache.

Who it’s for

Researchers and developers working on video world models, 3D scene generation, and efficient spatial consistency in video synthesis.

Highlights

• Latent Memory: Persistent 3D scene context is maintained directly in latent space.
• No RGB Detour: Eliminates the need for repeated render-and-reencode cycles.
• Memory Lifecycle: Implements a full cycle of initialization, reading, denoising, and updating across generated chunks.
• Efficiency: Achieves significantly faster generation speeds (10.57x) and drastically lower 3D cache memory usage (55x lower).