tylerlum/simtoolreal

What it solves

SimToolReal addresses the challenge of dexterous tool manipulation, specifically enabling a robot to use various tools for different tasks without needing to be retrained for every new object or scenario (zero-shot manipulation). It focuses on creating a policy that is "object-centric," allowing it to generalize across different tool categories and instances.

How it works

The project uses Reinforcement Learning (RL) to train policies in simulation environments (Isaac Sim or Isaac Gym). It employs algorithms like PPO and SAPG to develop a policy that can be deployed both in simulation (Sim-to-Sim) and in the real world (Sim-to-Real). For real-world deployment, it integrates with a perception system (SAM + FoundationPose) to track the tool's pose and a goal pose node to guide the robot's actions.

Who it’s for

This is designed for robotics researchers and engineers working on dexterous manipulation, sim-to-real transfer, and generalizable robot policies.

Highlights

• DexToolBench: A specialized benchmark covering 6 tool categories (hammer, marker, eraser, brush, spatula, screwdriver) across multiple objects and tasks.
• Zero-Shot Capability: Aims for policies that can handle tool manipulation without task-specific fine-tuning.
• Dual Simulation Support: Compatible with both the modern Isaac Sim (via Isaac Lab) and the legacy Isaac Gym.
• Integrated Deployment Pipeline: Includes a structured approach for moving from simulation to real-world hardware using a multi-node system (policy, goal pose, perception, and robot nodes).