hokindeng/object-permanence

What it solves

This project addresses the challenge of "object permanence" in world models—the ability of an AI to understand that objects continue to exist even when they are hidden from view (occluded). It provides a comprehensive framework for generating synthetic data and training models to predict the future state of a scene based on an initial video clip.

How it works

The system is divided into two main components:

1. Data Factory: A set of 150 Blender-rendered task generators that create video-to-video samples. These samples consist of an input video (the simulation up to a split point), a target video (the simulation after the split point), a text prompt, and ground-truth trajectories.
2. Training Stack: A native-PyTorch implementation of NVIDIA Cosmos3-Nano optimized for AWS Trainium2. It uses Tensor Parallelism and FSDP2 to train the model on the generated synthetic data to improve its predictive capabilities regarding occluded objects.

Who it’s for

This is primarily for AI researchers and developers working on world models, video generation, and spatial reasoning in machine learning.

Highlights

• Extensive Synthetic Dataset: Includes 150 cognitively grounded task generators for creating high-quality training data.
• High-Performance Training: Optimized for AWS Trainium2 using native PyTorch, avoiding XLA graph tracing.
• Detailed Ground Truth: Provides per-frame trajectories for every object in the scene for precise evaluation.
• Integrated Pipeline: Covers the entire workflow from data generation via Blender to large-scale model training.