NVIDIA/Isaac-GR00T

NVIDIA Isaac GR00T N1.7 - A Foundation Model for Generalist Robots.

NVIDIA Isaac GR00T N1.7 – A Vision‑Language‑Action Model for Humanoid‑Scale Robots

What it is – GR00T (General‑purpose Robot‑learning Transformer) is an open‑source vision‑language‑action (VLA) model that takes video frames and natural‑language instructions and outputs continuous robot actions. It is built for cross‑embodiment use: the same weights can control many different manipulators, from bimanual arms to full‑body humanoids.

Why it matters – By training on a mix of robot demonstrations and 20 000 h of human egocentric video (EgoScale), the model learns manipulation priors that transfer from human motion to robot control. The newest GA release (N1.7) upgrades the visual backbone to the Cosmos‑Reason2‑2B / Qwen3‑VL foundation model and switches to a relative end‑effector (EEF) action space, which makes the policy more robust when applied to robots with different kinematics.

Key components

  • Backbone – Cosmos‑Reason2‑2B (a Qwen3‑VL vision‑language model) that encodes images at native aspect ratios.
  • Action head – A diffusion‑transformer (DiT) that denoises a sequence of continuous actions, now with a 40‑step horizon.
  • Data format – Uses the LeRobot v2 parquet‑based dataset with an extra modality.json that maps raw state/action vectors to named fields (e.g., joint angles, gripper). This lets the same code handle many robot morphologies.
  • Embodiment tags – A tag (e.g., LIBERO_PANDA, OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT) tells the model how to interpret the state/action layout for a given robot.

How you would use it

  1. Install – Clone with submodules, install git‑lfs, then set up the environment with the fast uv tool (or pip). GPU drivers (CUDA 12.8/13.x) and FFmpeg 4‑7 are required because video decoding relies on torchcodec.
  2. Run inference – Load a checkpoint (e.g., nvidia/GR00T‑N1.7‑3B) and call the Gr00tPolicy API, supplying a dataset and the appropriate --embodiment-tag. A notebook (GR00T_inference.ipynb) walks through a zero‑shot demo on the DROID sample data.
  3. Fine‑tune – Use launch_finetune.py with your own demonstrations (converted to the LeRobot format). You can mix multiple dataset paths and weight them via ds_weights_alpha.
  4. Evaluate – Open‑loop evaluation scripts compare predicted actions against ground‑truth; simulation benchmarks (RoboCasa, SimplerEnv) and real‑hardware tests are provided.
  5. Deploy – Export the model to ONNX or TensorRT with the supplied scripts, then plug Gr00tPolicy into any robot controller. The repo includes platform‑specific Dockerfiles for DGX Spark, Jetson Thor, and Jetson Orin.

Available checkpoints – The base 3 B‑parameter model (GR00T‑N1.7‑3B) works zero‑shot on the pre‑trained embodiments. Several finetuned checkpoints are provided for popular platforms: LIBERO (Franka Panda), DROID, and SimplerEnv (WidowX, Google Robot).

Documentation & support – The repo contains detailed guides for hardware requirements, data preparation, fine‑tuning, evaluation, and deployment. Commercial support and a stable Apache‑2.0 license are offered for production use.

Bottom line – GR00T N1.7 is a genuine, production‑ready AI model that bridges vision, language, and continuous robot control, enabling researchers and engineers to prototype and ship general‑purpose manipulation skills across many robot bodies.

Related

  • Project
    dexmal/opendm

    OpenDM is a Vision-Language-Action (VLA) model and framework for open-world robot control, enabling robots to follow open-ended instructions and handle complex, long-horizon tasks.

  • Project
    Robbyant/lingbot-vla-v2

    LingBot‑VLA 2.0 is an open‑source Vision‑Language‑Action foundation model for robots. It unifies many robot embodiments into a 55‑dim action vector, uses MoE layers and dual‑query distillation (depth + future video) to learn predictive dynamics, and is trained on ~60 k h of robot and egocentric video data. The repo provides conda‑based installation, scripts to download 6 B‑parameter checkpoints, tools for data preparation, post‑training on custom datasets, and evaluation on the RoboTwin benchmark and real robots. Reported results show noticeable gains over the 1.0 version on bimanual and mobile manipulation tasks.

  • Dispatch
    Qwen-VLA: Unifying Vision-Language-Action Modeling for Embodied Intelligence

    Qwen-VLA is a general-purpose Vision-Language-Action model that unifies robotic manipulation, vision-language navigation, and cross-embodiment control into a single generalist policy model.

  • Project
    MINT-SJTU/Evo-1

    Evo-1 is a lightweight Vision-Language-Action (VLA) model for robotic manipulation that translates visual and textual instructions into robot actions across various simulations and real hardware.

  • Project
    X-Square-Robot/wall-x

    An open-source framework for embodied foundation models that enables robots to perform complex physical manipulations by bridging vision-language understanding with action prediction.