NVIDIA Cosmos-H-Dreams： Bringing Real-Time Generative Simulation to Surgical Robotics

世界基础模型提供了一条不同的路径。它们不是手动创建每一个对象和物理交互，而是直接从同步的视频和机器人运动学中学习视觉动态。NVIDIA 的 Cosmos-H-外科模拟器：https://huggingface.co/nvidia/Cosmos-H-Surgical-Simulator 展示了这一方法，通过从初始场景和一系列机器人动作生成未来的手术视频。它可以实现比物理更快的评估，并在 Open-H-Embodiment 生态系统：https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Open-H-Embodiment 中生成合成数据。

今天，我们介绍下一步：Cosmos-H-Dreams：https://github.com/isaac-for-healthcare/Cosmos-H-Dreams，一种用于手术机器人实时、动作条件生成的模拟器。Cosmos-H-Dreams 将 Cosmos-H-外科模拟器的功能提炼为因果的、少步的学生模型，并通过 FlashDreams：https://github.com/NVIDIA/flashdreams——NVIDIA 加速的流式推理库来提供服务。在单个 NVIDIA RTX PRO 6000 GPU 上运行，其结果是一个交互式环境，用户或学习到的策略可以在闭环中进行控制。

Cosmos-H-外科模拟器：https://huggingface.co/nvidia/Cosmos-H-Surgical-Simulator 是一个基于动作条件的世界基础模型，构建于 NVIDIA Cosmos-Predict2.5-2B：https://huggingface.co/nvidia/Cosmos-Predict2.5-2B，并在 Open-H-Embodiment：https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Open-H-Embodiment 数据集上进行后训练。在给定手术上下文帧和未来机器人轨迹的情况下，它生成显示这些动作可能视觉后果的视频。这使其对于离线策略评估和合成数据生成非常有用。记录的或策略生成的轨迹可以发送给模型，相应的回放可以生成，并且结果可以在不需重复在物理机器人上执行动作的情况下进行检查或评分。

Cosmos-H-Dreams 将模型推入实时模式。从 Cosmos-H-Surgical-Simulator 学到的多体现外科先验开始，我们针对 da Vinci Research Kit (dVRK) 桌面缝合任务对模型进行专门化，并将其蒸馏为一个因果学生模型，可以自回归地生成场景。发布的模型接收初始 RGB 帧和机器人运动学的实时流，然后生成下一段帧，再继续执行下一个动作块。

我们还通过与 CMR Surgical 和 Cambridge Consultants 合作，将 Cosmos-H-Dreams 与 Versius 外科医生控制器整合，展示了其多功能性，从而实现了在 Versius 平台上的实时操作。

关键挑战在于在降低生成成本的同时保留有用的手术动态。Cosmos-H-Dreams 使用为长时间、自回归 rollout 设计的教师-学生训练管线。

双向教师模型从 Cosmos-H-Surgical-Simulator Open-H 检查点开始，该检查点使用统一的 44 维动作表示。对于发布的 dVRK 桌面模型，双臂 dVRK 动作内容，包括末端执行器的相对平移、旋转和夹爪状态，被映射到这一通用表示中。

然后教师模型在 JHU dVRK 桌面混合数据上进行微调，包括成功演示以及失败和分布外场景，例如针掉落、投掷失败和结绑不成功。这些失败很重要：用于评估策略的模拟器必须重现错误操作的后果，而不仅仅是理想演示。

为了在长时间 rollout 中提高稳定性，训练过程中逐步增加教师模型的时间跨度。我们从 12 帧的时间跨度开始训练，并逐步增加到 72 帧。在每次跨度增加时，我们使用预训练权重初始化已预热的模型。

教师模型的去噪轨迹首先被预计算并缓存。因果学生模型从教师模型初始化，并接受训练以模仿这些缓存轨迹。这个预热阶段教会学生模型在开始从自身生成的历史中学习之前，如何使用因果注意力和流式键/值缓存进行操作。

自回归模型面临一个熟悉的问题：在训练期间，它们可能看到干净的、真实的上下文，而在部署期间，它们必须基于自身不完美的输出进行条件预测。因此，小错误可能会随着时间累积。

Cosmos-H-Dreams 通过自强制蒸馏 (self-forcing distillation) 解决了这种不匹配：https://arxiv.org/abs/2506.08009。在训练期间，学生模型使用其自己生成的上下文进行前滚。来自冻结教师模型的分布匹配监督会引导这些自生成的 rollout 向真实的外科视频靠拢。这为学生模型在交互式推理中将遇到的相同条件做好准备。

由此产生的模型支持少步骤扩散，每个潜在帧仅需两步去噪，而不是完整教师模型使用的多步过程。它将教师模型的外科先验知识与流式推理所需的因果结构结合起来。

模型蒸馏只是实时性能的一部分。Cosmos-H-Dreams 通过 FlashDreams 提供服务：https://github.com/NVIDIA/flashdreams，这是一种针对自回归世界和视频模型的加速推理库。

FlashDreams 通过几种互补优化将蒸馏后的学生模型转化为低延迟流式系统，例如流式 KV 缓存、CUDA 图捕获或模型编译。

这些技术共同将蒸馏后的外科世界模型微调，从标准 Cosmos-H-Surgical-Simulator 推理的大约每秒十帧的速度，提高到单个 NVIDIA RTX PRO 6000 上的交互操作速度（约每秒 160 帧）。

Cosmos-H-Dreams 还提供了将生成转化为交互的人机接口。浏览器客户端可以通过 WebRTC 发送键盘命令并接收生成的帧。Meta Quest 客户端可以将跟踪到的控制器动作映射为机器人动作，并通过 WebXR 显示合成场景。同一个模型也可以连接到学习到的外科策略，在闭环内交换生成的观测和预测的动作。

虽然Cosmos-H-Dreams包含一个预先训练的桌面缝合检查点，但该系统设计时可扩展到你具体的身体状态。为了训练你自己的数据集的实时学生模型，我们在逐步指南中提供了教师微调和自强制蒸馏的完整配方：https：//github.com/NVIDIA-Medtech/Cosmos-H-Surgical-Simulator/blob/main/docs/tutorial_teacher_training_and_self_forcing.md。

Cosmos-H-Dreams为外科模拟开辟了新领域：从真实机器人数据中学习到足够灵敏、能够被居住的环境。

接下来的紧迫步骤是评估不仅仅是视觉质量。一个有用的手术模拟器必须能正确响应动作，在长时间展开中保持器械和场景结构，并支持传达给物理机器人的结论。这激发了一系列新的闭环基准：工具提示的触达和姿势准确性、夹持器周期的真实度、空闲稳定性、反事实行动多样性、长视野漂移，以及模拟与真实政策结果的一致性。

实时世界模型也可以成为外科政策制定的积极合作伙伴。它们可以按需生成罕见的失败，提供可扩展的强化或模仿学习环境，并实现对新策略的快速评估，而无需将每个实验都绑在稀缺的机器人硬件上。

更远发展，实时仿真支持一系列下游应用，如延迟感知的远程手术，世界模型有助于保持更稳定的显示;或互动式手术排练、手术计划和术中决策支持。Cosmos-H-Dreams 是一个研发平台，而非诊断系统，也不是手术中影像的替代品，或是物理手术机器人的控制器。但它为安全探索这些可能性提供了基础。

随着模型保真度、时间稳定性和硬件效率的持续提升，实时生成仿真可以帮助将外科医生教育、合成数据生成、政策培训和政策评估连接到一个共享的物理人工智能环境中。

探索Cosmos-H-Dreams背后的模型、数据和运行时间：

Cosmos-H-Dreams 将动作条件的手术世界建模引入实时循环，为人类和策略提供了一个新的环境来练习、探索、生成数据以及评估下一步会发生什么。

World foundation models offer a different path. Instead of manually authoring every object and physical interaction, they learn visual dynamics directly from synchronized video and robot kinematics. NVIDIA's Cosmos-H-Surgical-Simulator：https://huggingface.co/nvidia/Cosmos-H-Surgical-Simulator demonstrated this approach by generating future surgical video from an initial scene and a sequence of robot actions. It enabled faster-than-physical evaluation and synthetic data generation across the Open-H-Embodiment：https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Open-H-Embodiment ecosystem.

Today, we are introducing the next step: Cosmos-H-Dreams：https://github.com/isaac-for-healthcare/Cosmos-H-Dreams, a real-time, action-conditioned generative simulator for surgical robotics. Cosmos-H-Dreams distills the capabilities of Cosmos-H-Surgical-Simulator into a causal, few-step student model and serves it through FlashDreams：https://github.com/NVIDIA/flashdreams, NVIDIA's accelerated streaming-inference library. Running on a single NVIDIA RTX PRO 6000 GPU, the result is an interactive environment that a person or a learned policy can control in a closed loop.

Cosmos-H-Surgical-Simulator：https://huggingface.co/nvidia/Cosmos-H-Surgical-Simulator is an action-conditioned world foundation model built on NVIDIA Cosmos-Predict2.5-2B：https://huggingface.co/nvidia/Cosmos-Predict2.5-2B and post-trained on the Open-H-Embodiment：https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Open-H-Embodiment dataset. Given a surgical context frame and a future robot trajectory, it generates video showing the likely visual consequences of those actions. This makes it useful for offline policy evaluation and synthetic data generation. A recorded or policy-generated trajectory can be sent to the model, the corresponding rollout can be generated, and the result can be inspected or scored without repeatedly executing the motion on a physical robot.

Cosmos-H-Dreams moves the model into the real-time regime. Starting from the multi-embodiment surgical priors learned by Cosmos-H-Surgical-Simulator, we specialize the model for da Vinci Research Kit (dVRK) tabletop suturing and distill it into a causal student that generates the scene autoregressively. The released model receives an initial RGB frame and a live stream of robot kinematics, then produces the next chunk of frames before continuing with the following action block.

We have also demonstrated the versatility of Cosmos-H-Dreams by collaborating with CMR Surgical and Cambridge Consultants to integrate it with the Versius surgeon controller, enabling real-time operation on the Versius platform.

The key challenge is preserving useful surgical dynamics while reducing the cost of generation. Cosmos-H-Dreams uses a teacher-to-student training pipeline designed for long, autoregressive rollouts.

The bidirectional teacher begins from the Cosmos-H-Surgical-Simulator Open-H checkpoint, which uses a unified 44-dimensional action representation. For the released dVRK tabletop model, the dual-arm dVRK action content, consisting of relative end-effector translation, rotation, and gripper state, is mapped into this common representation.

The teacher is then fine-tuned on the JHU dVRK tabletop mixture, including successful demonstrations as well as failure and out-of-distribution episodes such as needle drops, missed throws, and unsuccessful knot ties. These failures are important: a simulator intended to evaluate policies must reproduce the consequences of poor actions, not only ideal demonstrations.

To improve stability during long rollouts, the training process progressively increases the teacher's temporal horizon. We start training on a 12-frame horizon and progressively increase this number until 72 frames. At each horizon bump, we initialize the warmed-up model with pretrained weights.

The teacher's denoising trajectories are first precomputed and cached. A causal student is initialized from the teacher and trained to imitate these cached trajectories. This warmup stage teaches the student to operate with causal attention and a streaming key/value cache before it begins learning from its own generated history.

Autoregressive models face a familiar problem: during training they may see clean, ground-truth context, while during deployment they must condition on their own imperfect outputs. Small errors can therefore compound over time.

Cosmos-H-Dreams addresses this mismatch with self-forcing distillation ：https://arxiv.org/abs/2506.08009. During training, the student rolls forward using its own generated context. Distribution-matching supervision from the frozen teacher then guides those self-generated rollouts toward realistic surgical video. This prepares the student for the same conditions it will encounter during interactive inference.

The resulting model supports few-step diffusion, with as few as two denoising steps per latent frame, rather than the many-step process used by the full teacher. It combines the teacher's surgical priors with the causal structure needed for streaming.

Model distillation is only part of the real-time story. Cosmos-H-Dreams is served through FlashDreams：https://github.com/NVIDIA/flashdreams, an accelerated inference library for autoregressive world and video models.

FlashDreams turns the distilled student into a low-latency streaming system through several complementary optimizations, such as streaming KV cache, CUDA Graph capturing, or model compilation.

Together, these techniques bring the distilled surgical world model fine-tune from the roughly ten-frames-per-second regime of standard Cosmos-H-Surgical-Simulator inference to interactive operation (~160 frames per second) on a single NVIDIA RTX PRO 6000 .

Cosmos-H-Dreams also provides the human-machine interfaces that turn generation into interaction. A browser client can send keyboard commands and receive generated frames over WebRTC. A Meta Quest client can map tracked controller motion into robot actions and display the synthesized scene through WebXR. The same model can also be connected to a learned surgical policy, with generated observations and predicted actions exchanged inside a closed loop.

While Cosmos-H-Dreams includes a pre-trained checkpoint for tabletop suturing, the system is designed to be extensible to your specific embodiment. To train a real-time student model for your own dataset, we provide a complete recipe for teacher fine-tuning and self-forcing distillation in our step-by-step guide：https://github.com/NVIDIA-Medtech/Cosmos-H-Surgical-Simulator/blob/main/docs/tutorial_teacher_training_and_self_forcing.md.

Cosmos-H-Dreams opens a new frontier for surgical simulation: environments learned from real robot data that are responsive enough to be inhabited.

The immediate next step is to evaluate more than visual quality. A useful surgical simulator must respond correctly to actions, preserve instrument and scene structure over long rollouts, and support conclusions that transfer to the physical robot. This motivates a new family of closed-loop benchmarks: tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, counterfactual action diversity, long-horizon drift, and agreement between simulated and real policy outcomes.

Real-time world models can also become active partners in surgical policy development. They can generate rare failures on demand, provide scalable environments for reinforcement or imitation learning, and enable rapid evaluation of new policies without tying every experiment to scarce robotic hardware.

Further ahead, real-time simulation enables a series of downstream applications such as latency-aware telesurgery, where the world model helps maintain a more stable display; or interactive surgical rehearsal, procedure planning, and intraoperative decision support. Cosmos-H-Dreams is a research and development platform, not a diagnostic system, a replacement for intraoperative imaging, or a controller for a physical surgical robot. Yet it provides a foundation for exploring these possibilities safely.

As model fidelity, temporal stability, and hardware efficiency continue to improve, real-time generative simulation can help connect surgeon education, synthetic data generation, policy training, and policy evaluation within one shared Physical AI environment.

Explore the models, data, and runtime behind Cosmos-H-Dreams:

Cosmos-H-Dreams brings action-conditioned surgical world modeling into the real-time loop, creating a new environment for people and policies to practice, explore, generate data, and evaluate what happens next.