unrealcv/unrealcv

UnrealCV: Connecting Computer Vision to Unreal Engine

What it solves

UnrealCV provides a bridge between Unreal Engine (UE) and external AI frameworks like PyTorch or TensorFlow, allowing computer vision researchers to create and use virtual worlds as synthetic data sources or training environments for AI agents.

How it works

It functions as a plugin for Unreal Engine that enables communication between the game engine and an external program. It provides a set of specific commands to interact with the virtual world, control cameras, and manipulate objects. Users can either use a pre-compiled game binary with UnrealCV embedded or install the plugin into the engine to build their own custom environments.

Who it’s for

Computer vision researchers who need high-fidelity synthetic environments for training and testing AI models, particularly those focusing on perception and interaction.

Highlights

  • Engine Integration: Supports Unreal Engine 5.2 and later.
  • External Control: Allows external programs to programmatically control the virtual world via a command system.
  • Synthetic Data: Supports optical flow image capture and realistic rendering for dataset generation.
  • Blueprint Integration: Enables calling Blueprint functions directly from Python.
  • Flexible Deployment: Available as both a plugin for developers and a pre-built binary for non-engine users.

Related

  • Project
    prajwalshettydev/UnrealGenAISupport

    A plugin for Unreal Engine that simplifies the integration of LLMs and Generative AI, featuring support for multiple AI APIs and the Model Control Protocol for AI-driven editor control.

  • Project
    ChiR24/Unreal_mcp

    EngineWire is a native Unreal Engine plugin that lets LLM‑based assistants (Claude, Cursor, VS Code, etc.) control the editor through a single MCP tool called `unreal`. It exposes ~400 actions—spawning actors, editing Blueprints, building materials, running Play‑In‑Editor, taking screenshots, and more—via a simple search/describe/execute flow over either a local HTTP server (port 3000) or a Node.js stdio proxy (port 8090). Security is token‑based and localhost‑only by default. Install the plugin, enable the server, add the token to your AI client, and you can issue natural‑language commands that the plugin translates into Unreal‑Editor operations.

  • Project
    kevinpbuckley/VibeUE

    An MCP expansion and AI editor toolset for Unreal Engine 5.8+ that enables AI agents to perform deep world-building, animation, and performance profiling within the editor.

  • Project
    Classic298/open-webui-plugins

    A collection of plugins for Open WebUI that adds features like live interactive visualizations, vision capabilities for text-only models, and automated system maintenance.

  • Project
    opencv/opencv

    OpenCV is an open-source computer vision library that provides the tools and algorithms necessary to build AI-powered visual interpretation software.