AmyangXYZ/reze-mipo

What it solves

Reze MiPo is a real-time motion capture system that allows users to drive MMD (MikuMikuDance) characters using a webcam. It eliminates the need for complex configuration or specific T-poses, enabling users to record their movements—including full body, hands, and facial expressions—and export them as .vmd animation files for use in MMD software.

How it works

The system uses MediaPipe HolisticLandmarker running in a Web Worker to detect 3D points from a webcam feed. It then translates these points into MMD-specific bone rotations (quaternions) and morphs (blendshapes) through a custom solver.

Key technical implementations include:

• Automatic Calibration: The solver calibrates to a model's rest pose upon loading, allowing it to work with any PMX model without manual setup.
• Swing-Twist Decomposition: To reduce shimmering and noise, bone rotation is split into swing and twist components, with the twist receiving a dedicated filter.
• Advanced Filtering: It employs One-Euro filters for smoothing, a Savitzky-Golay pass for exported files to reduce jitter, and specific low-pass filtering for the Z-channel of landmarks.
• Collision Avoidance: It uses the model's own physics capsules to ensure limbs do not clip through the character's chest.
• Temporal Interpolation: It interpolates between captured frames to provide 60 FPS motion from a 30 Hz capture rate.

Who it’s for

Digital artists, MMD animators, and content creators who want a fast, browser-based way to capture human motion and translate it into high-quality MMD animations without expensive hardware.

Highlights

• Full-body capture: Tracks body, both hands (21 points each), and face simultaneously.
• Zero-config setup: Works with any PMX model by calibrating to the rest pose on load.
• Browser-based: Runs in the browser using WebGPU for rendering and MediaPipe for detection.
• VMD Export: Saves takes as .vmd files identical to those created on any machine.
• High-fidelity motion: Includes forearm twist, shoulder shrugging, and spinal curving for more natural movement.