ZhuoyangLiu2005/T-Rex

What it solves

T-Rex addresses the lack of dynamic tactile reactivity in robotic manipulation. While most Vision-Language-Action (VLA) models rely on static tactile encoders or ignore touch entirely, T-Rex enables robots to react in real-time to high-frequency tactile signals, which is essential for tasks requiring delicate force control and the manipulation of deformable objects.

How it works

• Asynchronous Mixture-of-Transformers (MoT): Built on a Qwen3-VL-2B backbone, the model uses three specialized experts (latent, action, and tactile) running at different frequencies. A "slow" action denoising process (5 Hz) handles general reasoning, while a "fast" tactile refinement process (20 Hz) allows the robot to adjust its movements within an action chunk without re-running the entire vision stack.
• Cascaded Flow Matching: This mechanism couples the slow and fast experts, enabling the policy to react to contact dynamically.
• Temporal Tactile VQ-VAE: A 1-D convolutional VQ-VAE tokenizes high-frequency force and deformation data over time, allowing the model to process tactile signals as discrete tokens.
• Three-Stage Training: The system is trained via large-scale tactile-free pretraining, followed by tactile-reactive midtraining, and finally task-specific post-training.

Who it’s for

• Robotics researchers and engineers working on dexterous manipulation.
• Developers building VLA models for contact-rich tasks.
• Users of the Dexmate Vega-1 or similar bimanual robotic systems.

Highlights

• Tactile-Reactive Dataset: Includes a 100-hour dataset (50 hours open-sourced) featuring 22 motor primitives and 200+ objects.
• High-Frequency Response: Achieves a 30% higher average success rate than strong baselines across 12 contact-rich tasks.
• Embedded Tokenizer: The tactile VQ-VAE can be embedded directly into the model for on-the-fly encoding of raw tactile data.
• Complete Stack: Provides the full data-collection stack, including teleoperation code using Manus gloves and VIVE trackers.