sail-sg/envpool

C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.

What it solves

EnvPool addresses the performance bottlenecks associated with Reinforcement Learning (RL) environment parallelization. Traditional Python subprocess-based vector environments often suffer from high overhead, limiting the throughput of experience collection. EnvPool provides a high-performance, C++-based execution engine that significantly increases the number of frames or steps per second that can be processed during training.

How it works

EnvPool is implemented in C++ and uses pybind11 to provide a Python interface. It manages a pool of environments and utilizes a thread pool to execute them in parallel. It supports both synchronous execution (where the user waits for all environments to step) and asynchronous execution (using send and recv functions to interact with environments as they finish).

Who it’s for

It is designed for RL researchers and developers who need to generate large amounts of training data quickly across various environments, including Atari, MuJoCo, and others, and who want a drop-in replacement for standard Gymnasium or DeepMind dm_env APIs.

Highlights

  • Extreme Throughput: Capable of reaching 1 million Atari frames and 3 million MuJoCo steps per second on high-end hardware (DGX-A100).
  • Broad Environment Support: Integrates a wide array of environments including Atari, MuJoCo, Box2D, Procgen, and Google Research Football.
  • API Compatibility: Fully compatible with Gymnasium and dm_env APIs, making it easy to integrate with libraries like Stable-Baselines3, Tianshou, and CleanRL.
  • Cross-Platform: Supports Linux, macOS, and Windows.
  • Advanced Rendering: Built-in batched RGB output and human display modes, with native C++ pixel observations for MuJoCo.

Related

  • Project
    rl-tools/rl-tools

    A fast, portable C++ library for deep reinforcement learning and continuous control, optimized for everything from microcontrollers to GPUs.

  • Project
    tensorforce/tensorforce

    A modular TensorFlow-based deep reinforcement learning framework for research and practice, providing flexible components to build and deploy RL agents.

  • Project
    jatinchowdhury18/RTNeural

    A lightweight C++ neural network inferencing engine optimized for real-time systems and audio processing.

  • Project
    hosseinmoein/DataFrame

    A high-performance C++ library for in-memory tabular data analysis and transformation, providing a fast alternative to Pandas and Polars for large-scale statistical and ML datasets.

  • Dispatch
    mujoco-py 1.50.1.0 release notes / what's new

    OpenAI has open-sourced mujoco-py 1.50.1.0, a high-performance Python 3 binding for the MuJoCo engine that introduces batched simulation and GPU-accelerated rendering.