ToyBlocks: A controlled world for robot learning and evaluation

Overview of ToyBlocks

ToyBlocks is a new simulation framework designed to serve as a "FoundationWorld" for robotics research, providing a controlled environment where researchers can systematically vary training data, model designs, and test conditions. By focusing on a constrained domain—a robot arm manipulating blocks on a tabletop—ToyBlocks enables the study of complex robotics questions, such as how models generalize to new tasks and how to effectively combine high-level reasoning with precise physical control.

Core Framework Requirements

ToyBlocks is built to address the limitations of narrow, real-world robotics demonstrations by fulfilling four key requirements:

  1. Realistic physics and appearance: Ensuring simulated behaviors closely mirror physical counterparts.
  2. Automatic generation at scale: Creating tasks and demonstration data without the need for time-consuming human teleoperation.
  3. Reasoning and dexterity: Challenging models to combine high-level planning with physical manipulation skills.
  4. Broad domain coverage: Providing a comprehensive set of tasks within the defined environment.

Technical Architecture

ToyBlocks integrates several components to facilitate end-to-end robot learning:

  • Skill Generation: Uses reinforcement learning (RL) and motion planning within Nvidia's Isaac Lab to develop non-prehensile skills like pushing, flipping, and rotating blocks.
  • Task-and-Motion Planning (TAMP): Automatically sequences these skills to solve complex tasks, producing labeled demonstration datasets at scale—reaching up to 24 times real-time performance on a single GPU.
  • Hierarchical Control: The framework supports a split between high-level reasoning models (e.g., Molmo 2) that decide the strategy and low-level action models (e.g., ManiFlow) that execute the specific physical movements.

Research Applications

ToyBlocks provides a testbed for several critical areas of robotics research:

  • Spatial Generalization: Researchers can test how vision-language-action (VLA) models handle variations in object layouts, lighting, and workspace boundaries, helping to identify why many current models remain brittle.
  • Long-Horizon Planning: By creating multi-step tasks like spelling words with blocks, the framework allows for the evaluation of hierarchical planning, including scenarios where the agent must perform exploratory actions to reveal information.
  • Agentic Systems: The framework supports integration with frontier models via the Model Context Protocol (MCP), allowing agents to use natural language to interact with the environment, form hypotheses, and design their own experiments.
  • World Model Evaluation: ToyBlocks provides ground-truth state data, enabling researchers to measure how accurately action-conditioned world models (such as Wan 2.2 or LeWM) predict physical interactions and task outcomes.

Sim-to-Real Transfer

To bridge the gap between simulation and the real world, the framework employs system identification to tune simulated parameters—such as joint friction and inertia—to match physical robot trajectories. Furthermore, it uses domain randomization for physical and visual properties and generates clean depth maps from RGB images to improve inference. Initial experiments showed that hierarchical systems trained entirely in the ToyBlocks simulation achieved comparable success rates when deployed on a physical Franka FR3 robot, demonstrating the potential for simulation-based training to translate to real-world hardware.

Sources