← All model news

NVIDIA Cosmos 3 adds four open-weight checkpoints: driving multi-sensor and dual-arm simulation, but first clarify deployment prerequisites

According to an NVIDIA blog post dated November 5, 2026, the Cosmos 3 family adds four open-weight checkpoints post-trained from Cosmos 3 Nano, each with 16B trainable parameters, covering driving multi-sensor generation, streaming simulation, simulator rendering and dual-arm action prediction, under the OpenMDW1.1 license.

Published Updated
Cover from the original report — NVIDIA Cosmos 3 adds four open-weight checkpoints: driving multi-sensor and dual-arm simulation, but first clarify deployment prerequisites
Cover from the original reportOriginal report ↗

What changed

In a blog post dated November 5, 2026, NVIDIA announced four open-weight checkpoints in the Cosmos 3 family, with weights hosted in the Cosmos3 collection on Hugging Face, alongside a technical report and demo page. The four models are described as post-training examples built on Cosmos 3 Nano, each with 16B trainable parameters, under the OpenMDW1.1 license. The division of labor is as follows: Cosmos3-Nano-Transfer-Auto is an offline bidirectional model that generates synchronized video from up to 11 camera views (including fisheye) and supports joint RGB and LiDAR generation, with per-camera World Scene Maps (WSM) controlling scene structure and traffic motion and text prompts guiding appearance; Cosmos3-Nano-Sim-Auto is a streaming variant of Transfer-Auto after causal distillation, continuously producing synchronized camera and LiDAR chunks with few denoising steps, where closed-loop interaction requires a simulation runtime and a WSM renderer; Cosmos3-Nano-Sim-Transfer renders a physics simulator's spatial controls (depth, segmentation, edges, blur) plus text prompts into policy-consumable RGB, with physics and ground-truth state remaining in the simulator; Cosmos3-Nano-Sim-Bimanual directly predicts the next observation from scene and robot actions, supporting the YAM and AgiBot dual-arm embodiments. The post-training recipe has four stages: bidirectional adaptation, teacher forcing, consistency distillation, and self-forcing; self-forcing exposes training to histories generated by the student model itself, to mitigate error accumulation. The publisher's runtime requirements are Linux, an NVIDIA GPU, Docker and the NVIDIA Container Toolkit, and it states that the released vLLM-Omni configuration was tested on a single GB200, which is not a minimum VRAM specification.

What it means for developers

The first implication for developers is the integration form: this is not an API you can call by swapping a model name, but a self-hosted inference stack requiring Linux, an NVIDIA GPU, Docker and the NVIDIA Container Toolkit. The blog says the released vLLM-Omni configuration was tested on one GB200 and explicitly says this is not a minimum VRAM specification; the blog body gives no numbers for the VRAM floor, whether a single card suffices, or per-chunk generation time, so deployment budgets must be measured yourself. The two rendering paths have different prerequisites, and the trade-offs are concrete. Sim-Transfer leaves motion, contact and physics state to the simulator, and the model only renders spatial controls such as depth, segmentation, edges or blur plus text prompts into RGB; the benefit is that physics and ground-truth state stay on the simulator side, at the cost of needing a working simulator and these control channels first. Sim-Bimanual predicts the next observation directly from scene and robot actions, skipping the simulator, but you must align with its data contract: T transitions require T+1 observations, the mixed-action example uses 59 control values padded to 64 during training, and coordinate frames, rotations, gripper calibration and validity masks all have schema constraints, with released inference actions already normalized. The original text only lists these conventions and does not describe how outputs would distort or whether errors would be raised when they are violated, so format validation can only be treated as a step you must test yourself. Streaming generation determines the evaluation framing. The model runs on compressed latents; after the initial image, each latent time step represents four RGB frames; Transfer predicts one step per chunk, while the report's optimized Bimanual configuration predicts two. The smaller the chunk, the more frequently new controls affect generation. The blog specifically explains the role of self-forcing in the four-stage recipe: a slightly mispredicted object becomes context for subsequent frames, so errors can accumulate, meaning normal short-range frames do not mean long-horizon rollouts are usable. The base is a Mixture-of-Transformers that connects a separate text-processing network and a visual generation network via attention, with key-value cache retaining attention information for the next prediction; these structures mean offline synthesis and streaming rollout are two different engineering shapes whose throughput expectations cannot be shared.

Integration and evaluation

The following is conditional analysis based on the report, offered as recommendations rather than reported facts. Decision one: render with a simulator, or predict directly from actions. The report offers two checkpoints with different prerequisites: Sim-Transfer takes simulator-computed motion, contact and physics state as input and renders them into color observations via spatial controls and text prompts; Sim-Bimanual starts from one image and generates the next observation directly from actions. If you already have a trustworthy physics simulator and policy training needs photorealistic observations, choose Sim-Transfer, with cost mainly in preparing control channels; if the bottleneck is having no simulator and only collected dual-arm manipulation data, Sim-Bimanual is more direct, provided your embodiment is within YAM or AgiBot coverage, or you are willing to retarget actions per the schema. Measurable acceptance criteria: on held-out episodes, compare the geometric consistency between generated observations and input controls, for example the overlap between segmented regions in generated frames and the input segmentation control, and whether depth ordering matches the control; then compare the success rate on held-out episodes of policies trained on these generated observations against a baseline. Decision two: use the released Sim-Auto directly, or first post-train Transfer-Auto on your own driving data and then distill. The report states the path dependency clearly: Sim-Auto is a streaming variant obtained from Transfer-Auto via causal training and few-step distillation, and for new sensor configurations or domains the publisher recommends adapting Transfer-Auto first, then distilling to Sim-Auto. If your camera layout, LiDAR configuration and target environment match the checkpoint's coverage, using the released Sim-Auto for streaming rollout is the shorter path; once they diverge, the cost becomes a two-stage effort of four-stage post-training plus distillation, and you must keep full training checkpoints for resumption, because exported inference weights do not include the optimizer or other training state. Measurable acceptance criteria: first use small held-out episodes to verify training and restart work, noting the blog's hint that a short run only validates execution, not convergence; then measure drift in long-horizon rollout, for example whether camera and LiDAR stay synchronized after the Nth chunk in the same scene, and whether response latency after control changes stays stable. Decision three: whether to build closed-loop interaction now. The report says closed-loop interaction requires a simulation runtime and a WSM renderer to continuously update controls, and this runtime is not included in the weights. If you only need offline synthetic data, you can skip it; if the goal is to study how policies respond to controls, include the runtime and renderer in your engineering budget and use response latency after control changes as a quantitative metric rather than judging single-frame image quality alone.

Limits and open questions

On fidelity and applicability boundaries, the report itself gives no benchmark numbers. The blog says the next work is to measure where the models faithfully support policy learning and evaluation and to understand their limitations, meaning the publisher has not yet disclosed fidelity results usable for cross-comparison; the technical report and demo page are linked, but this round saw no third-party reproduction, and this article draws no conclusion on their quality. The base model and capability descriptions come from NVIDIA's own blog and are the publisher's own statements. On licensing, this only restates the original: the models use OpenMDW1.1, permitting commercial and non-commercial use under its terms. Weights being downloadable on Hugging Face does not mean they can be used under arbitrary conditions; specific rights require reading the license terms themselves. Availability needs to be explicit: Cosmos 3 and these four checkpoints have no corresponding entries in the current KimiSeek catalog, and this article does not list them as any online service; their weight acquisition, hosting and runtime follow NVIDIA's and Hugging Face's releases. The hardware floor is likewise unknown; the blog only says the vLLM-Omni configuration was tested on one GB200 and stresses this is not a minimum VRAM specification. The report date is November 5, 2026; if weights, license terms or runtime requirements are updated after that, the publisher's pages should take precedence.

Three minutes to your first working call

Create an account and an API key. Calls use your account balance. 30% off your first top-up. Top up ¥500, pay ¥350, receive ¥500 in credit. Top up ¥1000, pay ¥850, receive ¥1000 in credit You can still start with a ¥10 trial top-up.

Sources and editorial process

NVIDIA 发布面向交互式世界模拟的 Cosmos 3 新开放模型

AI assisted the analysis and translation of the cited sources, followed by automated quality checks. Recommendations are not test results. Consult the current catalog for availability and pricing.