spear-sim/spear

What it solves

SPEAR addresses the limitations of existing photorealistic simulators, which often suffer from limited generality, poor programmability, and slow rendering speeds. It provides a way to programmatically control any Unreal Engine (UE) application using Python, allowing researchers to create complex, deterministic simulations for embodied AI research.

How it works

SPEAR is a Python library that connects to Unreal Engine applications via a modular plugin architecture. It exposes over 14,000 unique UE functions to Python, allowing users to access C++ functions and variables visible to UE's reflection system. It uses a transaction-based programming model where work is defined within begin_frame and end_frame contexts to ensure deterministic execution within a single UE frame.

Who it’s for

It is designed for researchers in embodied AI, synthetic data generation, and robotics, who need high-fidelity, photorealistic environments with deep programmatic control over the scene and agents.

Highlights

• High-Performance Rendering: Renders 1920x1080 photorealistic images directly into NumPy arrays at 73 FPS.
• Extensive API: Exposes over 14K UE functions, significantly increasing programmable functionality compared to previous simulators.
• Advanced Sensor Modalities: Provides ground truth data including depth, surface normals, semantic IDs, and non-diffuse intrinsic image decomposition.
• Flexible Agent Control: Supports controlling diverse agents (humans, cars, robots) across various UE projects.
• Co-simulation Support: Capable of coordinating with external physics simulators like MuJoCo.
• Natural Language Interface: Supports agentic scene editing via natural language prompts through an AI coding assistant.