NVIDIA Isaac GR00T N1.5: Post-Training for LeRobot SO-101 Arm

NVIDIA has announced the availability of Isaac GR00T N1.5, the first major update to the GR00T N1 foundation model. This cross-embodiment model allows developers to process multimodal inputs—including language and images—to perform manipulation tasks across diverse environments, with the ability to be post-trained for specific robotic hardware, tasks, and environments.

Cross-Embodiment Adaptation via EmbodimentTag

Isaac GR00T N1.5 utilizes an EmbodimentTag system to enable seamless customization across different robotic platforms. This system allows the model to be adapted to hardware it was not originally pre-trained on, such as the affordable, open-source LeRobot SO-101 arm. By designating a robot as a new_embodiment, developers can fine-tune the foundation model's reasoning and manipulation capabilities to match their specific hardware configuration.

Post-Training Workflow for LeRobot SO-101

Adapting GR00T N1.5 to the SO-101 arm involves a structured pipeline from dataset preparation to physical deployment.

Dataset Preparation and Compatibility

Fine-tuning requires a compatible dataset, such as the so101-table-cleanup dataset. To make a dataset "GR00T-compatible," developers must configure a modality.json file that provides essential information about state and action modalities. For single-camera setups, a specific modality configuration is used, while dual-camera setups require a different configuration file.

Fine-Tuning Process

The model is fine-tuned using the scripts/gr00t_finetune.py script. Key technical considerations for training include:

  • VRAM Requirements: Default settings require approximately 25GB of VRAM. Users with lower VRAM can use the --no-tune_diffusion_model flag to reduce memory usage.
  • Training Parameters: Example configurations include using a single GPU, setting max-steps (e.g., 10,000), and specifying the data-config (e.g., so100_dualcam).

Evaluation and Deployment

Before physical deployment, the policy is evaluated in an open-loop setting using scripts/eval_policy.py. Once validated, the model is deployed as an inference server via scripts/inference_service.py, which then communicates with the physical robot arm through a client script. This allows the robot to execute tasks based on language instructions, such as "Grab pens and place into pen holder."

Community Insights and Technical Optimizations

Following the release, community members and NVIDIA authors have highlighted several optimization strategies for improving model performance:

  • Reducing Motion Jerk: To address jerky movements, NVIDIA suggests removing the --no-tune_diffusion_model flag to ensure the entire expert head is tuned and increasing the action horizon via data_config.py. Community users also recommend increasing denoising steps to 16 and setting an action horizon of 16 for complex tasks.
  • Dataset Versions: The current implementation primarily supports LeRobot dataset v2. Users with v3 datasets may need to convert them to v2 or use the LeRobot repository, which has integrated GR00T N1.5 and supports v3.
  • Generalization: Some users reported that the model may ignore text prompts and stick to a single action from the dataset if overtrained, suggesting a need to balance max-steps to avoid overfitting.

"I would suggest for anybody running this for complicated tasks, at least have denoising steps of 16 and also have like an action horizon of 16. That's the only time I managed to get good results."

Available Resources

  • Model: The nvidia/GR00T-N1.5-3B model is available on Hugging Face.
  • Code: Fine-tuning scripts and sample datasets are hosted in the Isaac-GR00T GitHub repository.
  • Datasets: Example datasets include youliangtan/so101-table-cleanup and youliangtan/so100_strawberry_grape.

Sources