NVIDIA Cosmos Reason 2 release notes / what's new

NVIDIA has released Cosmos Reason 2, an open reasoning vision-language model (VLM) designed to provide robots and AI agents with the common sense and physical reasoning necessary to plan and act in the physical world. The model is currently the top-ranked open model for visual understanding on the Physical AI Bench and Physical Reasoning leaderboards.

Technical Capabilities and Improvements

Cosmos Reason 2 improves upon its predecessor by enhancing spatio-temporal understanding and timestamp precision. It is designed for flexible deployment across edge and cloud environments, available in 2B and 8B parameter sizes.

Key technical upgrades include:

  • Expanded Context Window: The input token limit has increased to 256K, a significant jump from the 16K tokens supported by Cosmos Reason 1.
  • Enhanced Visual Perception: The model now supports 2D/3D point localization, bounding box coordinates, trajectory data, and Optical Character Recognition (OCR).
  • Deployment Flexibility: Support for various deployment scales via 2B and 8B model sizes.

Primary Use Cases for Physical AI

Cosmos Reason 2 is applied across three primary domains to enable more autonomous and methodical decision-making in physical environments.

Video Analytics AI Agents

Cosmos Reason 2 enables agents to extract insights from large volumes of video data. With the addition of OCR and 2D/3D point localization, these agents can interpret text within videos to assess environmental conditions. NVIDIA provides a Video Search and Summarization (VSS) blueprint to accelerate the development of these agents. For example, Salesforce uses the VSS blueprint and Cosmos Reason as the VLM to analyze footage from Cobalt robots for workplace safety and compliance.

Data Annotation and Critique

The model automates the generation of high-quality, time-stamped captions and detailed descriptions for training datasets. Uber is utilizing Cosmos Reason 2 to create searchable video captions for autonomous vehicle (AV) training data. In a co-authored recipe for AV Video Captioning and VQA, the 8B model showed measurable improvements after fine-tuning:

  • BLEU scores: Increased by 10.6% (0.113 to 0.125).
  • MCQ-based VQA: Gained 0.67 percentage points (80.18% to 80.85%).
  • LingoQA: Increased by 13.8% (63.2% to 77.0%).

Robot Planning and Reasoning

Cosmos Reason 2 serves as the reasoning engine for Vision Language Action (VLA) models. Beyond determining the next logical step in a task, it can now provide specific trajectory coordinates for robot grippers. Encord has integrated Cosmos Reason 2 into its Data Agent library to support robotics and physical AI developers.

The Cosmos Model Family Ecosystem

Cosmos Reason 2 is part of a broader suite of models designed for physical AI:

  • Cosmos Predict 2.5: A generative AI model that predicts future physical states as video. It is pre-trained on 200 million clips and can generate up to 30 seconds of physically consistent video based on text, image, or video inputs. It is available in 2B and 14B sizes.
  • Cosmos Transfer 2.5: A multicontrol model for video-to-world style transfer, used to scale simulations across different environments and lighting conditions, often paired with NVIDIA Isaac Sim or NVIDIA Omniverse NuRec.
  • NVIDIA GR00T N1.6: An open reasoning VLA model specifically for humanoid robots, which utilizes Cosmos Reason to enhance its contextual understanding and full-body control.

Sources