openaiotlab/CUHK-X

What it solves

CUHK-X addresses the lack of comprehensive, synchronized multimodal datasets for human activity recognition (HAR) and reasoning. It provides a large-scale resource to move beyond simple action classification toward complex human action understanding (HAU) and next-action reasoning (HARn), which are critical for healthcare monitoring and smart environments.

How it works

The project provides a dataset of 64,267 samples across seven synchronized modalities: RGB video, infrared (IR), thermal imaging, depth, mmWave radar, skeleton data, and IMU sensors. It organizes data into two categories: "Small Model Data" for singular action classification and "Large Model Data" for sequential activities. The repository includes PyTorch pipelines for small-model baselines and benchmark code for evaluating large vision-language models (VLMs) like QwenVL and Video-LLaVA on tasks such as action captioning, emotion analysis, and next-action prediction.

Who it’s for

Researchers and developers working on multimodal learning, sensor fusion, human activity recognition, and the evaluation of large language models (LLMs) in physical, real-world activity contexts.

Highlights

• Seven Synchronized Modalities: Combines visual (RGB, IR, Thermal, Depth), spatial (Skeleton), and non-visual (mmWave Radar, IMU) data.
• Reasoning Benchmarks: Includes specific tasks for sequential reordering, temporal reasoning, and causal inference of human intentions.
• LLM-Driven Annotation: Uses a prompt-based framework to create logical and spatio-temporal scene descriptions.
• Diverse Evaluation: Supports cross-trial, cross-subject (LOSO), and cross-domain performance analysis.