Dai-Wenxun/MotionLCM

MotionLCM – Real‑time Controllable Human Motion Generation

What it is

• MotionLCM is a research codebase that generates 3‑D human motion sequences from text prompts and optional control signals (e.g., a desired trajectory for a specific joint). It builds on a latent consistency model (a fast alternative to diffusion) applied to a motion‑latent diffusion backbone, enabling one‑step or few‑step inference and thus real‑time generation.

Key ideas

• Latent Consistency Model (LCM): replaces the many diffusion steps with a single (or a few) deterministic update, dramatically cutting runtime.
• Motion ControlNet: a conditioning network that injects explicit control cues (initial motions or joint trajectories) into the latent space, preserving controllability while keeping the fast inference.
• Motion VAE & Motion Latent Diffusion (MLD): the pipeline first learns a compact latent representation of human motion (VAE) and a diffusion model in that space; MotionLCM then fine‑tunes this diffusion model for rapid sampling.

What you can do with it

• Motion reconstruction – encode a ground‑truth motion into the latent space and decode it back, useful for evaluating representation quality.
• Text‑to‑motion – give a natural‑language description (e.g., "a person jumps and waves") and obtain a plausible 3‑D motion.
• Motion control – supply a sparse trajectory for a joint (pelvis, wrists, etc.) and let the model generate a full‑body motion that follows that path while respecting the text prompt.
• Render SMPL meshes – convert the generated joint trajectories into full SMPL body meshes and render them with Blender.

Quick start (high‑level)

1. Create a conda environment (python=3.10, install requirements.txt).
2. Install system tools – ffmpeg (for GIFs) and git‑lfs.
3. Download assets – GloVe embeddings, T5 text encoder, SMPL models, and pre‑trained checkpoints via the provided prepare/*.sh scripts.
4. Prepare the HumanML3D dataset (or use the tiny demo dataset). The repository expects the dataset layout shown in the README.
5. Run demos:

   # reconstruction
   python demo.py --cfg configs/vae.yaml
   # text‑to‑motion
   python demo.py --cfg configs/motionlcm_t2m.yaml --example assets/example.txt
   # motion control (with ControlNet)
   python demo.py --cfg configs/motionlcm_control_s.yaml
   

   Outputs (GIF + .pkl file) are saved under experiments_*_test/.../demo_<timestamp>.
6. Render SMPL – install chumpy, then run python fit.py --pkl assets/example.pkl to get mesh data, and use the bundled Blender scripts to produce images or videos.

Training your own models

• Update the YAML config files (configs/*.yaml) for the task you want (VAE, MLD, MotionLCM, ControlNet).
• Run the corresponding training modules, e.g.:

  python -m train_vae --cfg configs/vae.yaml
  python -m train_motionlcm --cfg configs/motionlcm_t2m.yaml
  

• Evaluation follows the same pattern with python -m test --cfg <config>.

Where it fits

• Part of the emerging field of text‑conditioned human motion synthesis, similar to works like MotionDiffuse, TEMOS, and OmniControl.
• By leveraging LCM, MotionLCM pushes the boundary toward interactive, real‑time applications (e.g., virtual avatars, game animation, robotics simulation) where latency matters.

Resources

• Paper (arXiv:2404.19759) – https://arxiv.org/abs/2404.19759
• Demo videos on YouTube / Bilibili (links in the badge bar).
• Hugging Face Space for an online demo.
• Project webpage with more visual results.

License

• Non‑commercial use only (see LICENSE). Dependent libraries retain their own licenses.

---

All details above are taken directly from the repository’s README; no additional features have been inferred.