liruiw/GenSim

Generating Robotic Simulation Tasks via Large Language Models

What it solves

GenSim addresses the difficulty of manually creating diverse and complex robotic simulation tasks. It automates the generation of simulation environments and expert goals, allowing researchers to scale the variety of tasks available for training and benchmarking multitask robotic policies.

How it works

The project uses a Large Language Model (LLM) code generation pipeline—specifically GPT-4 and fine-tuned versions of Code-LLaMA—to write the actual code for simulation environments. It employs several prompting strategies to guide the LLM, including bottom-up template generation, top-down task generation, and task-conditioned chain-of-thought generation. These generated tasks are then integrated into the CliPort framework for training and evaluating robotic agents.

Who it’s for

It is designed for robotics researchers and AI engineers working on multitask policy training and embodied intelligence who need a large volume of diverse simulation data to train more robust agents.

Highlights

  • Automated Task Creation: Uses LLMs to write simulation code and define expert goals automatically.
  • Diverse Prompting: Supports multiple generation modes like top-down and chain-of-thought to create different types of tasks.
  • Model Flexibility: Compatible with GPT-4 and supports fine-tuning Code-LLaMA for specialized task generation.
  • Policy Benchmarking: Includes over 100 generated tasks that can be used as a benchmark for multitask policy training.

Related

  • Project
    AgibotTech/genie_sim

    A simulation platform for embodied intelligence that provides tools for 3D scene reconstruction, LLM-driven environment generation, and automated benchmarking for robot policies.

  • Project
    marcelo-earth/generative-manim

    A suite of tools that uses LLMs to translate natural language descriptions into Manim animations, enabling anyone to create technical videos without programming skills.

  • Project
    NVIDIA-NeMo/Skills

    A collection of pipelines for LLM development that streamlines synthetic data generation, training, and evaluation across local workstations and large-scale GPU clusters.

  • Project
    Snowflake-Labs/agent-world-model

    A synthetic environment generation pipeline that creates thousands of SQL-backed tool-use environments for large-scale agentic reinforcement learning.

  • Project
    Farzad-R/LLM-Zero-to-Hundred

    A collection of LLM projects and tutorials covering RAG, agentic workflows, multimodal chatbots, and fine-tuning pipelines to move from basic to advanced AI application development.