yangtiming/Fast-SAM-3D-Body

What it solves

Fast SAM 3D Body addresses the high inference latency of the original SAM 3D Body (3DB) framework, which takes several seconds per image. This delay makes it impossible to use for real-time applications like humanoid robot control or interactive motion tracking.

How it works

It is a training-free acceleration framework that optimizes the 3DB inference pathway through several techniques:

• Parallelization: It decouples serial spatial dependencies to allow parallel multi-crop feature extraction and streamlined transformer decoding.
• Pruning: It applies architecture-aware pruning to reduce computational overhead.
• Hardware Acceleration: It utilizes torch.compile and TensorRT to optimize model execution.
• Neural Mapping: It replaces the slow, iterative mesh fitting process (MHR-to-SMPL) with a direct feedforward neural mapping, speeding up this specific step by over 10,000x.

Who it’s for

This project is designed for researchers and developers working on 3D human mesh recovery, computer vision, and robotics, specifically those needing real-time full-body tracking from a single RGB camera stream.

Highlights

• Significant Speedup: Achieves up to a 10.9x end-to-end speedup over the original SAM 3D Body.
• Real-Time Performance: Enables humanoid robot control with an end-to-end latency of approximately 65 ms per frame on an NVIDIA RTX 5090.
• High Fidelity: Maintains reconstruction accuracy comparable to the original 3DB and even surpasses it on certain benchmarks like LSPET.
• Robotics Integration: Demonstrated utility in vision-only teleoperation and the collection of manipulation policies for humanoid robots.