What Is World Modeling? ：#what-is-world-modeling Two Transformer Towers, One Shared Representation ：#two-transformer-towers-one-shared-representation A Common Representation for Action ：#a-common-representation-for-action Policy Mode ：#policy-mode Post-trained samples for improved performance ：#post-trained-samples-for-improved-performance Try Cosmos 3 Edge ：#try-cosmos-3-edge The real world is vast and to operate in it physical AI systems need to understand how a scene is changing, anticipate what could happen next, and determine how an action will affect the world.
Machines operate at the edge in factories, warehouses, hospitals and beyond. To understand and act in these environments, they need models that deliver data center–level performance on memory-constrained systems.
Today we are releasing NVIDIA Cosmos 3 Edge on Hugging Face’s Cosmos 3 repository. It’s a 4-billion-parameter open world model that helps robots and vision AI agents understand their surroundings, reason in real time and generate robot actions on edge devices.
Download the model: https://huggingface.co/nvidia/Cosmos3-Edge：https://huggingface.co/nvidia/Cosmos3-Edge
The model delivers memory-efficient, high-throughput inference across NVIDIA edge computers including NVIDIA RTX PRO GPUs, NVIDIA DGX, NVIDIA GeForce RTX™ GPUs, NVIDIA Jetson including the newly announced Jetson T2000 and T3000 modules.
Designed as a compact open model that can serve as a small vision language model (VLM) with best-in-class throughput and accuracy with real-time inference.
As a post-trained world action model (WAM), Cosmos 3 Edge operates at robot-control resolution (640×360 observations), delivering real-time reasoning and generating 32 actions per inference on NVIDIA Jetson Thor - while achieving real-time control at 15 Hz.
Among similar size (4B parameters) models, Cosmos 3 Edge ranks #1 on VANTAGE-Bench for vision analytics and state-of-the-art for robot policy learning, setting the state of the art for smart infrastructure and robotics.
A world model learns how an environment changes over time. It represents objects, motion, spatial relationships, and the effects of actions.
Consider a robot reaching for an object. Recognizing the object is only the first step. The robot must also understand where the object is, how its gripper is moving, what may happen when contact occurs, and which action is most likely to complete the task successfully.
A world model helps the robot reason about these relationships. It can predict the visual result of an action, infer the action that caused a change, or generate an action to produce a desired outcome.
Cosmos 3 Edge brings these capabilities together in one model, on-device. Its shared representation enables physical AI systems to understand the current world state, simulate possible futures, and connect those futures to actions.
Cosmos 3 combines two transformer towers:
The two towers maintain separate normalization layers and multilayer perceptrons. They share multimodal attention layers, which align information across language, video, audio, and action.
This design allows the model to reason about a scene before generating an output. Depending on the task, Cosmos 3 can produce reasoning tokens from the autoregressive tower or denoised video and action tokens from the diffusion tower.
The attention pattern is also adapted to each type of information. Language uses causal attention, where each token attends to the tokens that came before it. Diffusion tokens attend more broadly to the available context, supporting coherent prediction and generation.
Together, these components give the model a common representation of what is happening, what may happen next, and how an action could influence the outcome.
Physical systems describe actions differently. A vehicle may represent an action through ego pose and movement. A camera uses camera motion. A robot arm uses the pose of its end effector, while a hand or gripper also needs to represent grasp state.
Cosmos 3 maps these different embodiments into a common action representation.
Actions are encoded as compact geometric vectors that capture:
This creates a direct connection between control and the visual structure of the world. The model can associate changes in pixels with physical motion, spatial relationships, and control inputs.
As a result, generated video becomes more than a visual prediction. It can represent how the world is expected to change in response to an action. This gives developers training data grounded in motion, control, and cause and effect.
As a policy, Cosmos 3 predicts an action together with its expected visual consequence.
The model can generate what the system should do and simulate what is likely to happen next. This connects world modeling directly to robot policy training and evaluation.
Current state in, action and visual consequence out.
These modes enable one model to learn cause, effect, and policy. Action can flow through the model in both directions, allowing Cosmos 3 Edge to predict the effects of an action or infer an action from its effects.
Prompt: Pick up the banana and put it in the plate.
We are also releasing Cosmos 3 Edge Policy (DROID): A robot manipulation policy post-trained on the DROID dataset for pick-and-place tasks, with accompanying post-training scripts.
Developers can use a small cluster of H100 or NVIDIA DGX Station to efficiently fine-tune Cosmos 3 Edge for their target workloads before deploying to NVIDIA edge and accelerated computing platforms.
Alongside the base models, we're releasing reference post-trained checkpoints and training recipes to help developers adapt and optimize Cosmos 3 for their own applications.
Cosmos 3 is an open world foundation model platform. As models are post-trained with high-quality, domain-specific data, their accuracy continue to improve for specialized applications. Cosmos serves as the starting point for building domain-adapted world models using open frameworks. Post-training enables developers to improve model performance while preserving output quality. To demonstrate this workflow, we're also releasing the Cosmos 3 Super 4-Step Distillation checkpoint along with a post-training script, providing a faster reference implementation for 64B model that helps developers adapt Cosmos to their own domains and achieve better downstream performance.
Cosmos 3 Super 4-Step (Text-to-Image & Image-to-Video): These are distribution-matching distilled checkpoint and scripts that reduce diffusion from 35–50 denoising steps to just 4, delivering up to 25× faster inference while preserving image and video quality and fidelity.
These examples serve as reference implementations that developers can build on. Using the open training scripts, you can post-train Cosmos 3 Edge for custom robot policies or distill Cosmos 3 Super for faster image and video generation tailored to your application.
Cosmos 3 Edge connects understanding, prediction, simulation, and action through a shared world representation.Developers can deploy it closer to their sensors on-device. The model can be used as a reasoner or as an action generator.
Use open Cosmos Frameworks to customize and specialize the models.
We continue to advance Cosmos 3 for Physical AI, with upcoming improvements in interactive world generation, driving scenario simulation, and robotics policies.
We are also investing in faster inference and more efficient post-training across a broader range of hardware, reducing the time and compute required to build downstream models.
This includes optimizing Cosmos 3 checkpoints with open inference and optimization frameworks such as vLLM, with more optimizations and developer tooling coming soon.
情报判断
Aioga 编辑摘要
NVIDIA 在 Hugging Face 发布 Cosmos 3 Edge。这是一款四十亿参数的开放世界模型，面向机器人和视觉 AI 智能体，可在边缘设备上理解环境、实时推理并生成机器人动作。
背景分析
公开材料称，物理 AI 系统需要理解场景变化、预判后续状态并评估动作影响。工厂、仓库和医院等边缘场景又受内存限制，因此需要兼顾推理性能与资源效率的模型。
扫码直达文章详情