nvidia-cosmos/cosmos-transfer1

Cosmos-Transfer1 is a world-to-world transfer model designed to bridge the perceptual divide between simulated and real-world environments.

What it solves

Cosmos-Transfer1 provides a framework for controllable, multimodal world generation. It allows users to transform various input signals—such as depth maps, segmentation masks, edge detection, LiDAR, or HDMaps—into realistic video simulations. This is particularly useful for Physical AI applications, such as autonomous vehicle simulation or robotics, where developers need to generate high-fidelity visual data from structured inputs.

How it works

It uses a foundation model architecture that acts as a controllable generator. Users provide a text prompt alongside one or more conditional inputs (e.g., a depth map or segmentation video). The model uses a ControlNet-based approach to guide the generation process, ensuring the output video adheres to the spatial and temporal constraints provided by the input modalities. It also includes an upscaler to convert 720p outputs to 4K resolution and supports distillation to speed up inference by reducing the number of diffusion steps required.

Who it’s for

This project is designed for Physical AI researchers and engineers who need to build, train, or customize world models for simulation, robotics, or autonomous vehicle development. It is intended for those who need to generate synthetic data or simulate environments based on specific sensor or structural inputs.

Highlights

  • Supports multi-modal conditional generation using combinations of depth, segmentation, edge, blur, LiDAR, and HDMap inputs.
  • Includes a 4K upscaler to enhance generated video resolution.
  • Provides scripts for post-training, pre-training from scratch, and model distillation for faster inference.
  • Features specialized support for autonomous vehicle (AV) applications, including single-to-multi-view generation.
  • Integrates Llama Guard 3 for content safety and moderation.

Related

  • Project
    iLearn-Lab/VLA-Diffusion-Policy-Robotics

    This project is a comprehensive survey of diffusion policies for robotic manipulation, systematically organizing the fast-growing literature by data representation, model architecture, and diffusion strategy. It provides a curated list of papers with links to help researchers quickly navigate and understand the field.

  • Project
    UM-ARM-Lab/pytorch_kinematics

    A parallel and differentiable robotics kinematics library for PyTorch that provides fast forward and inverse kinematics and Jacobian calculations.

  • Project
    RobotWebTools/roslibjs

    A collection of JavaScript client libraries that enable web applications to communicate with ROS (Robot Operating System) systems for building robot interfaces.

  • Project
    LeCAR-Lab/BFM-Zero

    BFM-Zero is a promptable foundation model for humanoid robot control that uses unsupervised reinforcement learning to enable versatile behaviors like motion tracking and goal-reaching from a single trained policy.