SimWorld-AI/SimWorld

What it solves

SimWorld provides a realistic, open-ended simulation platform for developing and evaluating LLM/VLM (Large Language Model/Vision-Language Model) agents. It addresses the need for complex physical and social environments where embodied AI can be tested on tasks like navigation, planning, and strategic cooperation without the risks or costs of real-world deployment.

How it works

Built on Unreal Engine 5, the platform uses a three-layer architecture:

1. Unreal Engine Backend: Handles physics simulation, rich assets, and the environment.
2. Environment Layer: Manages procedural city generation, language-driven scene editing, and provides Gym-like APIs for agent interaction.
3. Agent Layer: Allows LLM/VLM agents to process multimodal observations (RGB, depth, segmentation) and execute actions via a local action planner.

Users can start with a base package of city scenes or expand their simulation with over 100 pre-built maps and custom .pak files for their own environments and assets.

Who it’s for

It is designed for researchers and developers working on Embodied AI, autonomous agents, and the multimodal reasoning of LLMs and VLMs.

Highlights

• Multimodal Perception: Supports RGB images, depth maps, and segmentation masks for agent observations.
• Procedural Generation: Features language-based procedural city generation to create diverse scenarios.
• Gym-like Interface: Provides a familiar API for training and evaluating agents in a simulation.
• Customizable: Allows users to import their own Unreal Engine environments, assets, and agent models.
• Open-ended World: Combines accurate physics with social reasoning scenarios for complex behavior testing.