xavierpuigf/virtualhome
API to run VirtualHome, a Multi-Agent Household Simulator
What it solves
VirtualHome provides a simulated environment for complex household activities, allowing researchers to generate synthetic data and train agents for embodied AI tasks without needing a physical robot in a real home. It solves the problem of data scarcity and safety risks associated with training AI agents to interact with domestic environments.
How it works
The platform uses a combination of a Unity-based simulator for visual rendering and a Python API to execute "programs" (sequences of actions). It features two primary simulation modes:
- Unity Simulator: Renders high-fidelity videos and snapshots of activities based on Python scripts.
- Evolving Graph: A lightweight, Python-only simulator that tracks the state of the environment as a graph, representing how objects and their relationships change over time.
Who it’s for
It is designed for researchers in embodied AI, reinforcement learning, and computer vision who need a controlled environment to simulate human-like activities and generate ground-truth data (such as semantic segmentation and optical flow).
Highlights
- Interactive Environment: Supports complex interactions like picking up objects and operating appliances.
- Procedural Generation: Capable of creating an infinite variety of unique household environments.
- RL Integration: Includes OpenAI Gym-like environments and compatibility with Ray for parallel training.
- Ground-Truth Streaming: Provides time-stamped actions, depth, and instance/semantic segmentation.
- Multi-Agent Support: Allows for the simulation of activities involving multiple agents.