NVlabs/SimFoundry

Modular and Automated Scene Generation for Policy Learning and Evaluation

What it solves

SimFoundry automates the creation of physics-ready simulation scenes from short real-world videos. It eliminates the need for manual annotation and tedious 3D modeling when building digital twins for robotics research, allowing users to convert a video of a tabletop or room into a functional simulation environment in under an hour.

How it works

The system uses a modular pipeline consisting of three main stages:

  1. Reconstruction: A 13-stage process that takes a video and performs object segmentation, depth estimation, 2D-to-3D mesh generation, and pose estimation to compile a scene for the OmniGibson simulator.
  2. Augmentation: Uses Vision Language Models (VLMs) to propose and generate "digital cousins"—variations of the original objects in terms of geometry, appearance, and topology—and suggests potential manipulation tasks.
  3. Application: Integrates the reconstructed scene into OmniGibson for robot policy evaluation, teleoperation, and testing.

Who it’s for

It is designed for robotics researchers and developers who need high-fidelity simulation environments for training and evaluating robot policies, specifically those focusing on sim-to-real transfer.

Highlights

  • Fully Automated: Converts real video to simulation without manual annotation.
  • Modular Design: Components are independently swappable, allowing the pipeline to improve as new foundation models are released.
  • Digital Cousins: Automatically generates diverse object variations to improve policy generalization.
  • Physics-Ready: Outputs scenes complete with physical parameters and USD/OmniGibson exports.
  • Integrated Editors: Includes both a browser-based light editor for quick composition and an OmniGibson-based editor for physics-accurate adjustments.

Related

  • Project
    DevYangJC/Argus

    Argus is an open‑source, enterprise‑grade RAG platform that ingests private documents, builds both vector and keyword indexes, and answers questions with a LLM that must cite real sources. It includes a Spring AI‑based ReactAgent for chat, hybrid retrieval with RRF fusion, multi‑level evidence scoring, role‑based security, and a Vue 3 front‑end.

  • Project
    stack-of-tasks/tsid

    A C++ library for optimization-based inverse-dynamics control of rigid multi-body robots, such as humanoids and quadrupeds.

  • Project
    GREAT-WHU/MASt3R-Fusion

    A SLAM system that integrates feed-forward pointmap regression with IMU and GNSS data for high-accuracy, large-scale 3D mapping and perception.

  • Project
    aws-deepracer-community/deepracer-for-cloud

    A toolkit for running AWS DeepRacer training environments on cloud VMs or local computers, enabling independent, scalable reinforcement learning for autonomous racing.