sail-sg/envpool

C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.

What it solves

EnvPool addresses the performance bottleneck in reinforcement learning (RL) caused by slow environment simulation. Traditional Python-based vector environments often suffer from high overhead due to subprocesses, limiting the throughput of experience collection. EnvPool provides a high-performance, C++-based execution engine that allows for massive parallelization of environments with significantly higher frames-per-second (FPS) than standard Python implementations.

How it works

EnvPool is implemented in C++ and uses pybind11 to provide a Python interface. It manages a pool of environments and utilizes a thread pool to execute them in parallel. It supports two primary interaction modes:

  • Synchronous API: Closely follows Gymnasium and dm_env APIs, where step and reset functions operate on batches of environments by default.
  • Asynchronous API: Splits the step function into send and recv operations, allowing the user to feed actions and retrieve states without waiting for the entire batch to complete, further increasing efficiency.

It integrates with XLA via JAX jit functions and is compatible with major RL libraries like Stable-Baselines3, Tianshou, and CleanRL.

Who it’s for

It is designed for reinforcement learning researchers and developers who need to collect large amounts of experience quickly to train agents, particularly those using computationally expensive environments like Atari, MuJoCo, or various robotics simulators.

Highlights

  • Extreme Throughput: Capable of reaching ~1M Atari FPS and ~3M MuJoCo steps per second on high-end hardware (DGX-A100).
  • Broad Environment Support: Includes Atari, MuJoCo, Box2D, DeepMind Control Suite, ViZDoom, and more.
  • Cross-Platform: Available for Linux, macOS, and Windows.
  • Flexible APIs: Compatible with Gymnasium and dm_env; supports both sync and async execution.
  • Built-in Rendering: Supports batched rgb_array output and human display mode.
  • C++ Extensibility: Provides a developer API for integrating custom C++ environments.

Related

  • Project
    rl-tools/rl-tools

    A fast, portable C++ library for deep reinforcement learning and continuous control, optimized for everything from microcontrollers to GPUs.

  • Project
    jatinchowdhury18/RTNeural

    A lightweight C++ neural network inferencing engine optimized for real-time systems and audio processing.

  • Dispatch
    mujoco-py 1.50.1.0 release notes / what's new

    OpenAI has open-sourced mujoco-py 1.50.1.0, a high-performance Python 3 binding for the MuJoCo engine that introduces batched simulation and GPU-accelerated rendering.

  • Project
    Artelnics/opennn

    A high-performance C++ library for implementing neural networks used in advanced analytics for business, healthcare, and engineering.

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.