NVIDIA Cosmos3-H-Dreams: Generative Simulation for Clinical Surgery

NVIDIA Cosmos3-H-Dreams introduces interactive generative simulation to the visual and procedural complexity of clinical surgery, achieving the most convincing in-body imagery to date by learning visual dynamics directly from real clinical procedures.

From Tabletop Tasks to Clinical Scenes

In-body surgery requires a generative simulator to capture complex, dynamic responses that go beyond simple background movement, making temporal consistency essential for realistic simulation.

In a controlled tabletop environment, instruments operate against a simple background; however, inside the human body, small instrument movements trigger large-scale changes, such as tissue stretching under traction, shifting reflections, structure occlusion, and changes in smoke visibility.

To function as an interactive environment, the generated scene must maintain fidelity by preserving anatomy and instruments while following specific controls. This means that translation, rotation, and gripper commands must produce coherent anatomical consequences, not just plausible-looking video.

Cosmos3-H-Dreams addresses the requirement for low latency generation in this complex setting by utilizing a model specialized for clinical cholecystectomy and distilled for real-time operation, enabling action-conditioned interaction within scenes featuring complex anatomy, tissue motion, and occlusion.

Key Technical Advancements in Cosmos3-H-Dreams

Cosmos3-H-Dreams represents a significant leap from previous tabletop simulators by training on real-world clinical data and optimizing the model for real-time performance.

Foundation and Control Structure

Cosmos3-H-Dreams is built upon a specialized bidirectional teacher model, which is a refinement of Cosmos3-H-Embodied-Nano. This foundation is a 16-billion-parameter model pretrained on the Open-H-Embodiment dataset.

This model specializes in clinical cholecystectomy by focusing on two visible, engaged instruments: a left Cadiere grasper and a right monopolar hook. The model supports 28 action dimensions, which include pose and gripper information for both arms, along with energy controls, thumbstick axes, and thumbstick buttons.

Teacher-to-Student Training Pipeline

The system employs a sophisticated teacher-to-student training pipeline designed to handle long autoregressive rollouts while maintaining fast, interactive generation.

  1. Teacher Training: The bidirectional teacher model is first adapted on 24 future frames, and then the learning horizon is extended to 72 frames (2.4 to 7.2 seconds at 10 Hz) to allow the teacher to learn comprehensive surgical dynamics.
  2. Distillation: Distillation converts the bidirectional teacher into a causal student model capable of generating video in short blocks as new actions arrive. Causal teacher forcing prepares the student for sequential generation using recorded context.
  3. Error Mitigation: Self-Forcing trains the student on its own generated context, guided by the teacher and a learned critic. This process is crucial for mitigating the accumulation of errors during interactive rollouts.

The first released student checkpoint uses four denoising steps per generated block, while the second release checkpoint uses a two-step schedule. The two-step model is trained separately, allowing developers to balance visual quality against inference cost.

Real-Time Streaming Inference

To achieve interactive performance, Cosmos3-H-Dreams integrates FlashDreams, NVIDIA’s accelerated streaming-inference library, and an adapted FadeMem module.

FlashDreams provides the accelerated streaming runtime, and the FadeMem module consolidates the growing key/value cache during long rollouts. This cache consolidation complements Self-Forcing, addressing how the model's history is stored during the streaming generation process.

Cosmos3-H-Dreams runs interactively on a single NVIDIA RTX PRO 6000 GPU.

Capabilities and Generalization

Cosmos3-H-Dreams is designed to generalize beyond its training data, allowing it to produce coherent frames even when conditioned on unseen movements.

The simulator demonstrates that instrument control leads to distinct continuations of the same scene, even when starting from the same initial frame, illustrating physically accurate tissue deformations. These rollouts allow researchers to inspect whether instruments follow controls, whether tissue responds coherently to interaction, and whether anatomical and instrument appearance remains consistent over time.

Furthermore, the system is adaptable to new datasets and embodiments. A complete recipe is provided for fine-tuning and Self-Forcing distillation on custom data, provided the new controller uses the same action conventions and normalization as the training data.

Future Research Directions and Limitations

Cosmos3-H-Dreams is explicitly a research simulator and is not a validated system for patient care. The current capabilities open several critical research directions:

  • Procedural Expansion: Expanding coverage and temporal consistency to model the full cholecystectomy procedure systematically, defining clear task boundaries and success criteria.
  • Hybrid Modeling: Combining learned visual realism with explicit geometry, contact, and tissue mechanics from numerical simulation. Numerical models could constrain and validate deformation, making the simulation behavior inspectable.
  • Clinical Validation: Assessing anatomical accuracy, instrument behavior, and tissue response through expert feedback from surgeons, which could then be used to create reproducible benchmarks.
  • Advanced Interaction: Moving beyond two controlled tools to support richer coordination, occlusion, and interaction among multiple instruments.
  • Autonomy: Coupling world action models to Cosmos3-H-Dreams in a closed loop to allow policies to learn defined cholecystectomy subtasks.

Getting Started

Developers can explore the models, data, and runtime for Cosmos3-H-Dreams using the following resources:

  • Code and Examples: The full GitHub repository for Cosmos3-H-Dreams.
  • Model Checkpoints: The official Hugging Face checkpoint for Cosmos3-H-Dreams.
  • Adaptation Guide: A step-by-step guide detailing the process for teacher fine-tuning and Self-Forcing distillation on custom datasets.
  • Core Components: Access to Open-H-Embodiment dataset, FlashDreams runtime, and NVIDIA Cosmos 3 repositories.

Sources