squall01337/mixamo-llm-mocap

What it solves

This project allows users to convert locked-camera videos (either filmed or AI-generated) into clean Forward Kinematics (FK) animations for any Mixamo character. It eliminates the need for expensive motion capture suits or manual keyframing, specifically handling complex movements like fighting or multi-person interactions while ensuring the resulting animation is scriptable and compatible with standard Mixamo workflows.

How it works

The pipeline uses a pose estimator (GVHMR) to recover SMPL-X joint rotations from a video plate. It then retargets these rotations onto a Mixamo rig, adjusting for the specific proportions of the chosen character. The system employs several refinement techniques:

• Foot Planting: Uses a combination of detectors, physics, and floor constraints to prevent "foot skate" and ensure feet stay above the floor.
• Hand Tracking: Crops and zooms into hands to detect fists and open palms, as the base estimator lacks finger data.
• Trajectory Recovery: Recovers the static camera's position to remove height and horizontal drift.
• Multi-Person Support: Can split a single video into separate tracks for two performers, retargeting them onto different characters and verifying their spatial relationship and mesh collisions in Blender.
• QA Loop: A numerical evaluation gate checks for bone explosions, hip pops, and fidelity against the original video before final rendering.

Who it’s for

• 3D Animators: Those using Mixamo characters who want to automate the motion capture process from video.
• AI Content Creators: Users generating AI video that they wish to convert into controllable 3D animations.
• Game Developers: Developers needing quick, high-fidelity FK animations for characters with different proportions.
• AI Agents: The pipeline is designed with CLI and socket calls, making it operable by AI agents.

Highlights

• Character Agnostic: Works with any Mixamo character by measuring bone lengths and rest poses via a setup script.
• Pure FK Output: Produces quaternions at 30 fps, ensuring compatibility with most Mixamo-style workflows without needing cleanup.
• Numerical QA: Replaces "vibes-based" review with a numerical fidelity check that measures limb and head angles against the video keypoints.
• Multi-Character Interaction: Specifically handles two-person fights, managing separation and collision detection between different character meshes.