zju3dv/MotionStreamer

What it solves

MotionStreamer addresses the challenge of generating human motion sequences from text descriptions in a streaming fashion. It enables the continuous generation of motion, allowing for more fluid and seamless transitions between different actions described by text.

How it works

The system uses a diffusion-based autoregressive model operating within a causal latent space. The pipeline involves several stages:

1. Causal TAE: A Causal Temporal Autoencoder (TAE) is used to encode motion into a causal latent space, ensuring that the latent representations are based only on past and current frames.
2. Motion Generation: The model utilizes Llama blocks, a "Two-Forward" strategy, and QK-Norm to generate motion latents from text embeddings (provided by sentence-T5-XXL).
3. Streaming Support: By training on processed streaming datasets (like BABEL-stream), the model learns to generate motion sequences that are continuous and consistent across subsequences.

Who it’s for

Researchers and developers working in computer vision, 3D human motion synthesis, and character animation for games or virtual reality.

Highlights

• Streaming Generation: Capable of producing continuous motion sequences based on text inputs.
• Causal Latent Space: Uses a Causal TAE to ensure temporal consistency and prevent leakage of future information during generation.
• No IK Needed: The 272-dimensional motion representation used eliminates the need for Inverse Kinematics (IK) during recovery.
• Integrated Evaluation: Includes a TMR-based motion evaluator to measure quality via metrics like FID and Diversity.