Jianghanxiao/PhysTwin

PhysTwin – Physics‑informed reconstruction & simulation of deformable objects from video

What it is

• An open‑source research framework that takes RGB‑D video of a soft or deformable object (cloth, rope, plush toys, etc.) and builds a digital twin – a physics‑based model that can be simulated and interacted with.
• The twin is learned by combining visual cues with a differentiable spring‑mass simulator, so the recovered model respects physical constraints (forces, material stiffness, collisions).
• Once built, the twin can be explored in an interactive playground, visualised, or used for downstream tasks such as robot planning, RL, or XR/VR simulations.

Key capabilities

Capability	What you get
Reconstruction	Convert raw RGB‑D recordings into a physics‑informed deformable‑object model (the “PhysTwin”).
Interactive simulation	Play with the twin in real‑time (keyboard or remote control) via a Gradio‑based web UI or a local window.
Analysis tools	Visualise inferred forces, material parameters, and multi‑object collisions.
Optimization pipelines	Two‑stage training: a fast zero‑order (CMA‑ES) stage followed by a first‑order gradient‑based stage using Nvidia Warp for rapid convergence (≈5 min on a RTX 4090 for simple cases).
Extensibility	Example code for Model‑Predictive Path Integral (MPPI) planning, batched simulation (Boba), and robot‑physics integration (Real2Sim‑Eval).

Installation

• The repo supports Linux, Windows, Docker, and a special RTX 5090‑focused environment. The typical Linux flow is:

  conda create -n phystwin python=3.10
  conda activate phystwin
  bash ./env_install/env_install.sh          # installs core deps
  bash ./env_install/download_pretrained_models.sh
  

• Optional components (Trellis, Grounding‑SAM‑2, RealSense, SDXL) can be skipped if you only want the interactive playground.
• Docker users can build with ./docker_scripts/build.sh and run the playground inside the container.

Data & pretrained assets

• Pre‑processed datasets and model checkpoints are hosted on Hugging Face. Download the zip files (data.zip, experiments_optimization.zip, experiments.zip, gaussian_output.zip, optional additional_data.zip) and unpack them into the repository root.
• The data include multi‑view RGB‑D recordings, calibration files, and ready‑to‑run optimization results for several demo objects (rope, cloth, “sloth” plush, etc.).

Typical workflow

1. Process raw video (if you have your own recordings):

   python script_process_data.py
   python export_gaussian_data.py   # prepares first‑frame appearance
   

2. Optimize / train the twin:

   # Zero‑order (fast, coarse)   
   python script_optimize.py
   # First‑order (refine)        
   python script_train.py
   

3. Run inference / render:

   python script_inference.py
   bash gs_run.sh                 # Gaussian‑splatting rendering
   

4. Explore interactively:

   python interactive_playground.py --case_name double_lift_cloth_3 --n_ctrl_parts 2
   

   • Add --inv_ctrl to flip control direction.
   • Use the gradio_playground branch for headless‑server web access.
5. Analyse (force, material, multi‑object collisions) with the provided visualisation scripts, e.g.:

   python visualize_force.py --case_name single_push_rope_1
   python visualize_material.py --case_name double_lift_cloth_1
   

Demo highlights

• Boba (batched simulator) can run >3 300 FPS on an RTX 4090, enabling RL policy training in minutes.
• Deform360 dataset supplies 198 everyday objects with 1 980 interaction sequences captured from 41 cameras and tactile grippers.
• Real‑time interactive playground runs at ~37 FPS on a RTX 4090 for the “sloth” case.

Who might use it

• Researchers building soft‑body world models for robotics or VR/AR.
• Developers needing a pipeline to turn real video of deformable items into simulatable assets.
• Anyone interested in physics‑aware video understanding, differentiable simulation, or data‑driven soft‑body control.

Citation If you use PhysTwin in research, cite the accompanying ICCV 2025 paper (arXiv 2503.17973). The README also lists downstream works that have built on PhysTwin.

---

All details above are taken directly from the repository’s README; no external assumptions have been added.