NVIDIA Cosmos-H-Dreams: Real-Time Generative Simulation for Surgical Robotics

TL;DR

NVIDIA has released Cosmos-H-Dreams, a real-time, action-conditioned generative simulator for surgical robotics. By distilling the Cosmos-H-Surgical-Simulator into a causal student model and utilizing the FlashDreams inference engine, NVIDIA has enabled interactive, closed-loop simulation of surgical scenes at 160 frames per second (FPS) on a single GPU.

Transitioning from World Models to Interactive Simulators

Cosmos-H-Dreams evolves the previous Cosmos-H-Surgical-Simulator—an action-conditioned world foundation model based on NVIDIA Cosmos-Predict2.5-2B and trained on the Open-H-Embodiment dataset. While the original simulator was primarily used for offline policy evaluation and synthetic data generation, Cosmos-H-Dreams moves these capabilities into the real-time regime.

Specifically, the model is specialized for da Vinci Research Kit (dVRK) tabletop suturing. It takes an initial RGB frame and a live stream of robot kinematics as input to produce the next sequence of frames autoregressively. NVIDIA has also demonstrated the system's versatility by integrating it with the Versius surgeon controller from CMR Surgical and Cambridge Consultants.

The Distillation Pipeline for Real-Time Performance

To achieve real-time speeds without sacrificing surgical dynamics, Cosmos-H-Dreams employs a teacher-to-student training pipeline designed for long, autoregressive rollouts.

The Surgical Teacher

The process begins with a bidirectional teacher model based on the Cosmos-H-Surgical-Simulator Open-H checkpoint. This teacher uses a unified 44-dimensional action representation. For the dVRK tabletop model, dual-arm action content (relative end-effector translation, rotation, and gripper state) is mapped to this representation. The teacher is fine-tuned on the JHU dVRK tabletop mixture, which includes both successful demonstrations and failures (e.g., needle drops and missed throws) to ensure the simulator can accurately reproduce the consequences of poor actions.

To maintain stability during long rollouts, the teacher's temporal horizon is progressively increased from 12 frames to 72 frames during training.

Causal Warmup and Self-Forcing Distillation

The distillation process occurs in two primary stages:

1. Causal Warmup: A causal student is initialized from the teacher and trained to imitate precomputed and cached denoising trajectories. This teaches the student to use causal attention and a streaming key/value (KV) cache.
2. Self-Forcing Distillation: To prevent the compounding of errors that occurs when autoregressive models condition on their own imperfect outputs, NVIDIA uses self-forcing distillation. The student generates its own context, and a frozen teacher provides distribution-matching supervision to guide these rollouts toward realistic video.

This resulting student model supports few-step diffusion, requiring as few as two denoising steps per latent frame.

FlashDreams: The Real-Time Inference Engine

Cosmos-H-Dreams is served via FlashDreams, an accelerated inference library for autoregressive world and video models. FlashDreams implements several optimizations to reduce latency, including:

• Streaming KV cache
• CUDA Graph capturing
• Model compilation

These optimizations increase inference speed from approximately 10 FPS (standard Cosmos-H-Surgical-Simulator) to approximately 160 FPS on a single NVIDIA RTX PRO 6000 GPU. This enables several interaction modes:

• Browser Client: Keyboard commands and generated frames delivered via WebRTC.
• Meta Quest Client: Tracked controller motion mapped to robot actions via WebXR.
• Closed-Loop Policy: Integration with learned surgical policies where observations and actions are exchanged in real-time.

Future Directions in Surgical Physical AI

NVIDIA views Cosmos-H-Dreams as a foundation for closed-loop surgical Physical AI. Future development focuses on moving beyond visual quality to evaluate functional accuracy through new benchmarks, including:

• Tool-tip reach and pose accuracy
• Gripper-cycle fidelity
• Idle stability
• Counterfactual action diversity
• Long-horizon drift
• Agreement between simulated and real policy outcomes

Potential downstream applications include latency-aware telesurgery, interactive surgical rehearsal, procedure planning, and intraoperative decision support. NVIDIA notes that Cosmos-H-Dreams is a research and development platform and is not intended as a diagnostic system, a replacement for intraoperative imaging, or a physical robot controller.