Friedrich-M/UniMate

UniMate – One Unified Model to Animate Diverse Skeletons

What it is

A research‑grade deep‑learning model that generates 3‑D articulated motion from a text prompt for any rigged skeleton (humans, animals, robots, even abstract objects).
Trained on the UniML3D dataset – 13.8 k text‑paired motion clips covering 7.4 k skeletons of many topologies (biped, quadruped, avian, marine, insectoid, serpentine, rigid‑body rigs).

Why it matters

Existing text‑to‑animation systems are limited to a single character topology. UniMate learns a canonical representation of joints and can be applied to unseen rigs without any per‑skeleton fine‑tuning or test‑time optimisation.
The model works in real‑time, making it suitable for interactive applications (games, VR, robotics simulation).

Key components

Component	What it does
Model architecture	Two attention variants are provided: `graph_adaln` (spatial‑temporal graph‑aware attention) and `full_cross_attn` (standard transformer over joint‑time tokens). Both can be conditioned on text via a frozen FLAN‑T5 encoder.
Training objective	Flow‑matching diffusion: the network predicts the velocity of a linear interpolation between noise and the target motion. Loss combines masked L2, a geodesic rotation term, and a velocity‑smoothness term.
Data pipeline	`data_process/` scripts turn raw assets (Mixamo, Objaverse‑XL, Truebones) into the unified UniML3D format: export, render, caption, joint‑annotation, feature extraction. A `rig_preprocess` tool lets you bring any GLB/FBX/GLTF rig into the same format, with LLM‑generated joint labels for review.
Inference	`unimate.inference.sample` drives a trained checkpoint on a rig and a prompt, outputting animated GLB/FBX files. Supports batch sampling, classifier‑free guidance (`--cfg_scale`), and multiple repetitions per prompt.
Utilities	Pre‑compute text embeddings, resume training, visualise debug renders, and a simple script to drive meshes with generated motions.

Getting started

Create the environment

conda create -n unimate python=3.10 -y
conda activate unimate
pip install "setuptools<81"
pip install -r requirements.txt --no-build-isolation

Download the dataset (≈ 88 GB) and the released checkpoint:

hf download Linzhan/UniML3D --repo-type dataset --local-dir dataset
hf download Linzhan/UniMate --repo-type model --local-dir outputs \
    --include "unimate_uniml3d_f60_v3/*.json" \
    --include "unimate_uniml3d_f60_v3/*.npy" \
    --include "unimate_uniml3d_f60_v3/checkpoints/checkpoint_step_150000.pt"

Run a quick demo (animate a few pre‑processed assets):

python -m unimate.inference.sample \
    --exp_dir outputs/unimate_uniml3d_f60_v3 \
    --test_cases_json assets/examples/examples.json \
    --num_repetitions 3 \
    --output_dir outputs/samples/examples
SKIP_INVALID=1 bash scripts/run_animate_motion.sh outputs/samples/examples

You will get animated GLB/FBX files for objects like a Garfield model, a Gundam, a shark, etc.

Training your own model

Config files live in configs/. They encode dataset choice, frame length, attention type, and caption conditioning (e.g., uniml3d_60frames_graph_adaln.json).

Launch with 🤗 Accelerate:

accelerate launch -m unimate.training.train --config configs/uniml3d_60frames_graph_adaln.json

Multi‑GPU: add --num_processes N.
The pipeline handles balanced sampling across very uneven skeleton counts, joint‑axis padding, and data augmentations (joint addition, leaf removal, chain pooling, bone‑length perturbation).
Checkpoints, logs, and visual debug renders are written under outputs/<experiment>/.

Using your own rig ("Bring your own rig")

Place a GLB/FBX/GLTF file in a folder.
Run the preprocessing tool:
```
python -m data_process.rig_preprocess.process <path_to_folder>
```
The tool labels joints with an LLM, shows you a review UI for joint names and the facing direction, and produces the canonical asset directory expected by the inference script.
Animate with the same sample command, pointing --asset to the newly created directory.

Applications

Game/VR content creation – quickly generate diverse motion clips for characters or props without hand‑authoring animations.
Robotics simulation – produce plausible locomotion or manipulation sequences for robots with arbitrary kinematic trees.
Film & VFX prototyping – explore motion ideas for fantastical creatures or mechanical rigs.
The repository also demonstrates three zero‑shot downstream tasks (e.g., motion editing, style transfer) using the same model, though details are in the paper.

Links & resources

Project page & paper: https://linzhanmou.com/unimate/ (arXiv: 2609.05415)
Interactive demo: https://linzhanmou.com/unimate/interactive.html
Dataset: https://huggingface.co/datasets/Linzhan/UniML3D
Model checkpoints: https://huggingface.co/Linzhan/UniMate
GitHub repo: https://github.com/Friedrich-M/UniMate (this README)

TL;DR

UniMate is an open‑source diffusion‑based text‑to‑animation model that works on any rigged 3‑D skeleton. It comes with a large, multi‑topology dataset, full training code, a preprocessing pipeline for custom assets, and ready‑to‑run inference scripts. Perfect for researchers and developers who need a general‑purpose, real‑time motion generator.