liangpan99/TokenHSI

What it solves

TokenHSI addresses the difficulty of creating physics-based characters that can perform a wide variety of human-scene interaction (HSI) tasks. Instead of training separate models for every single action, it provides a unified way to synthesize diverse interactions, such as climbing, sitting, and carrying objects, while remaining adaptable to new environments and object shapes.

How it works

The project uses a single transformer network that employs "task tokenization" to unify multiple foundational HSI skills. This allows the model to learn a general set of interaction skills and then adapt those skills to more complex scenarios. The system is trained using reinforcement learning (via rl_games) and simulated in the IsaacGym environment, utilizing SMPL body models for humanoid representation.

Who it’s for

This is primarily for researchers and developers in robotics, computer animation, and embodied AI who are working on physically plausible human motion synthesis and humanoid control.

Highlights

• Unified Model: Combines multiple foundational skills (path-following, sitting, climbing, carrying) into one transformer network.
• Skill Composition: Can combine learned skills to perform complex tasks, such as carrying an object while following a path or climbing.
• Adaptive: Handles variations in object shapes (e.g., switching from carrying a box to a chair) and terrain shapes.
• Long-horizon Tasks: Capable of completing extended tasks in complex, dynamic environments.