unrealcv/unrealcv

What it solves

UnrealCV provides a bridge between Unreal Engine (UE) and external AI frameworks like PyTorch or TensorFlow, allowing computer vision researchers to create and use virtual worlds as synthetic data sources or training environments for AI agents.

How it works

It functions as a plugin for Unreal Engine that enables communication between the game engine and an external program. It provides a set of specific commands to interact with the virtual world, control cameras, and manipulate objects. Users can either use a pre-compiled game binary with UnrealCV embedded or install the plugin into the engine to build their own custom environments.

Who it’s for

Computer vision researchers who need high-fidelity synthetic environments for training and testing AI models, particularly those focusing on perception and interaction.

Highlights

• Engine Integration: Supports Unreal Engine 5.2 and later.
• External Control: Allows external programs to programmatically control the virtual world via a command system.
• Synthetic Data: Supports optical flow image capture and realistic rendering for dataset generation.
• Blueprint Integration: Enables calling Blueprint functions directly from Python.
• Flexible Deployment: Available as both a plugin for developers and a pre-built binary for non-engine users.