liangnjupt/VisTouch

A large-scale synchronized vision–touch–audio dataset of robotic sliding contact

What it solves

VisTouch provides a large-scale, synchronized dataset of robotic sliding contact to help AI models learn to recognize materials and understand the relationship between vision, sound, and touch. It addresses the lack of public datasets that strictly align these three modalities for robotic interaction.

How it works

Using a robot arm and a dexterous hand, the system records simultaneous video, audio, and force-tactile trajectories as the hand slides across eight different materials (such as brass, silk, and wood). The data is strictly synchronized using a shared wall clock, ensuring that every frame of video, sound sample, and tactile reading is aligned to the same event.

Who it’s for

Researchers and developers working on multimodal learning, robotic perception, and haptic feedback systems who need high-quality, paired data for training models.

Highlights

  • Strict Synchronization: The first public dataset to provide event-level synchronization across vision, sound, and force-tactile signals.
  • Diverse Materials: Includes 10,498 synchronized clips covering eight distinct materials.
  • Multimodal Benchmarks: Includes baseline models for material recognition, tactile super-resolution, and cross-modal generation (e.g., generating tactile force from sound).
  • Comprehensive Hardware: Data captured using a 2K camera, professional microphone, and a dexterous hand's integrated pressure/force sensors.

Related

  • Project
    rerun-io/rerun

    A multimodal data layer and visual debugger for physical AI, allowing developers to log, query, and visualize multi-rate sensor data in real-time.

  • Project
    google-research/language-table

    A suite of human-collected datasets and a benchmark for open vocabulary visuolinguomotor learning, enabling robots to execute a vast array of natural language instructions.

  • Project
    tudelft-iv/view-of-delft-dataset

    A multi-sensor automotive dataset containing synchronized LiDAR, camera, and 3+1D radar data with 3D bounding box annotations for road user detection in urban traffic.

  • Project
    InternRobotics/VLAC

    A Vision-Language-Action-Critic model that provides dense reward feedback and trajectory quality screening for real-world robotic reinforcement learning.

  • Project
    NVIDIA-ISAAC-ROS/isaac_ros_visual_slam

    A ROS 2 package that provides GPU‑accelerated stereo visual‑inertial SLAM for NVIDIA Jetson (and x86_64) platforms, delivering high‑rate odometry and loop‑closed maps for robots and drones.