blendi-remade/prompt2rig

What it solves

Creating 3D character animations usually requires motion capture equipment, manual keyframing, or searching through large motion libraries. prompt2rig removes that friction: you type a movement description, a video model generates a clip of someone performing it, and your GPU recovers clean 3D body and hand motion from that clip. The result plays live on a Mixamo character in the browser and can be exported to Blender or Unity. It also solves the usual pose-estimation problems like jitter, limb stretching, or twisted forearms by using a whole-clip model (GVHMR) that outputs joint rotations on a fixed-proportion body.

How it works

1. Prompt to video – You type a prompt (or pick a preset). A fast video model on fal (FLUX 3 Max for quality, H3 Max Turbo for speed) generates a short clip of a performer acting out the motion. Every clip starts from the same generated performer image, which pose estimation prefers.
2. Live preview – While the clip is being generated, MediaPipe Pose runs in the browser and retargets the pose per frame to the Mixamo rig, giving an instant rough preview that covers the wait.
3. High-quality motion pass – On your local GPU, GVHMR processes the whole clip at once and outputs SMPL body rotations plus a gravity-aware root path. In parallel, WiLoR estimates palm and all 15 finger joints per hand (fallback to MediaPipe Hands if you don't provide the MANO hand model). This pass takes about 3.5–4 seconds for a 5-second clip on an RTX 4090.
4. Retarget and export – The motion is retargeted to any Mixamo character (X Bot, Michelle, Soldier, or your own .fbx/.glb). You can chain clips in stream mode, swap characters, and export the skinned character with animation as a .glb file.

Who it's for

• Animators and game developers who want quick motion ideas from text prompts without manual rigging.
• Blender or Unity users who need a fast pipeline from text to an animated character.
• Creators exploring AI-assisted animation; the tool is tested on Windows 11 with an NVIDIA GPU.
• Tinkerers comfortable with Node.js and Python who want to run a local motion worker.

Highlights

• Prompt-only workflow – No motion capture equipment or manual keyframing; just describe the motion.
• Whole-clip motion recovery – GVHMR sees the entire clip, avoids jitter and limb artifacts, and recovers a gravity-aware root path.
• Finger tracking included – WiLoR adds palm and all 15 finger joints per hand, with a natural curl fallback when hands aren't detected.
• Live preview during generation – MediaPipe in the browser gives instant feedback while the HQ motion is computed.
• Mixamo compatibility – Works with any Mixamo rig (T-pose) and exports skinned .glb files suitable for Blender or Unity.
• Stream mode – Chains clips into one continuous performance; each clip picks up from the last frame.
• Fast and optimised – The HQ worker is tuned for single-person generated clips (10s → 3.4s per clip on an RTX 4090).
• Extras – Headless Blender rendering, off-screen capture tool, and a hand-pose test bench.
• License warning – The web app and live preview are Apache-2.0, but the HQ motion path (GVHMR, SMPL-X, WiLoR, MANO) is research/non-commercial; a commercial product must replace or license those pieces.