apple-aiml-research/ml-hypersim

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

📚 What is the Hypersim Dataset & Toolkit?

Hypersim is a large, photorealistic synthetic dataset aimed at holistic indoor scene understanding. It provides:

77 400 rendered images (≈1.9 TB) of 461 indoor scenes, each with dense per‑pixel ground‑truth labels.
Complete geometry, materials, lighting, camera poses, and intrinsics for every image.
Decomposed image channels – color, diffuse reflectance, diffuse illumination, and a residual term – so researchers can study shading, albedo, and illumination separately.
Rich semantic and instance annotations (NYU‑40 class labels) plus 9‑DOF 3‑D bounding boxes for every object.
A toolkit built on V‑Ray that lets you generate similar datasets, edit scenes, script camera trajectories, and add custom lens effects.

The project is a genuine research‑grade resource for computer‑vision and graphics work such as depth estimation, semantic segmentation, material estimation, and neural rendering.

🔑 Core Components

Component	What it contains	Why it matters
Image files (`scene_cam_XX_final_hdf5`)	HDR color, diffuse illumination, diffuse reflectance, residual (no tone‑mapping)	Enables learning of intrinsic image decomposition and physically‑based rendering tasks
Geometry files (`scene_cam_XX_geometry_hdf5`)	Depth (meters), world‑space positions, surface normals (with/without bump), semantic/instance IDs, render‑entity IDs, texture coordinates	Provides ground‑truth for depth, normal, and segmentation models
Metadata (`_detail/…`)	Camera trajectories (positions & orientations), scene scale, object‑to‑node mapping, per‑scene CSVs	Lets you reconstruct exact camera poses and convert asset units to meters
Mesh annotations (`mesh_objects_*.hdf5`, `metadata_objects.csv`)	Per‑object NYU‑40 class and instance IDs, object names	Supplies the semantic labels needed for segmentation and instance‑aware learning
Bounding‑box data (`metadata_semantic_instance_bounding_box_*.hdf5`)	9‑DOF (position, orientation, size) boxes for each instance	Useful for 3‑D object detection and pose estimation
Toolkit code (`code/python/tools/…`)	Scripts for downloading, tonemapping, visualising bounding boxes, generating new trajectories, etc.	Gives a ready‑made pipeline to create new synthetic data or adapt existing scenes

🎯 Typical Use‑Cases

Training data for indoor depth‑estimation, surface‑normal prediction, and semantic/instance segmentation networks.
Research on intrinsic image decomposition – the separate reflectance/illumination channels let you test algorithms that recover albedo and lighting.
Neural rendering / inverse graphics – the full V‑Ray scene information (materials, lighting, geometry) supports learning‑based rendering pipelines.
Benchmarking – the provided train/val/test split (v1) enables reproducible evaluation across papers.
Dataset generation – the Hypersim Toolkit can be used to synthesize new scenes or augment existing ones with custom camera paths or lens distortions.

🚀 Getting Started

Download the data

python code/python/tools/dataset_download_images.py \
    --downloads_dir /path/to/downloads \
    --decompress_dir /path/to/evermotion_dataset/scenes

The full release is ~1.9 TB; you can also use the community‑contributed script in contrib/99991 to fetch subsets.

Install the Python dependencies (Anaconda quick‑start):
```
conda env create -f environment.yml   # pulls in h5py, numpy, pandas, etc.
conda activate hypersim
```
(The README continues with detailed system‑level prerequisites – V‑Ray Standalone/AppSDK are required only if you plan to render new scenes.)

Explore the data

import h5py, numpy as np
f = h5py.File('ai_001_001/images/scene_cam_00_final_hdf5/frame.0000.color.hdf5', 'r')
color = np.array(f['data'])   # HDR color image

Use the provided scene_generate_images_tonemap.py to apply a tone‑mapping operator for visualisation.

Run a demo – the repository ships example scripts such as:
- scene_generate_images_bounding_box.py – renders an image with 3‑D instance boxes overlaid.
- scene_generate_images_tonemap.py – converts raw HDR channels to displayable PNGs.

📄 License & Citation

Dataset – Creative Commons Attribution‑ShareAlike 3.0 (CC‑BY‑SA 3.0).
Code – See the repository’s LICENSE file (mostly permissive, with some GPL‑linked components that can be omitted).

Citation – If you use Hypersim, cite the ICCV 2021 paper:

@inproceedings{roberts:2021,
    author = {Mike Roberts and Jason Ramapuram and Anurag Ranjan and Atulit Kumar and
              Miguel Angel Bautista and Nathan Paczan and Russ Webb and Joshua M. Susskind},
    title  = {{Hypersim}: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding},
    booktitle = {International Conference on Computer Vision (ICCV) 2021},
    year = {2021}
}

🛠️ The Hypersim Toolkit (Brief)

Low‑Level Toolkit – Direct manipulation of individual V‑Ray scene files (add geometry, set custom lens models, export per‑pixel ground truth).
High‑Level Toolkit – Batch operations over many scenes (auto‑generate collision‑free camera paths, apply semantic labels, orchestrate large‑scale renders).
Dependencies – V‑Ray Standalone/AppSDK (commercial), plus open‑source libraries (NumPy, h5py, OpenCV, etc.).
Disclaimer – Some optional components pull in GPL‑licensed code; the core toolkit remains usable without them.

TL;DR

The Hypersim Dataset gives researchers a massive, richly annotated indoor synthetic image collection, and the accompanying Toolkit lets you create or extend such data using V‑Ray. It’s a solid, research‑grade resource for any project that needs high‑quality ground‑truth for depth, normals, semantics, or intrinsic image decomposition.