cleanrl: a collection of high-quality single-file deep reinforcement learning implementations for research and prototyping

What it solves

CleanRL addresses the complexity and opacity of modular Deep Reinforcement Learning (DRL) libraries. Many RL libraries use heavy abstraction and subclassing, making it difficult for researchers to understand the exact implementation details of an algorithm or to prototype new features without navigating a deep hierarchy of files.

How it works

Instead of a modular architecture, CleanRL provides high-quality, single-file implementations of DRL algorithms. Every detail of a specific algorithm variant is contained within one standalone Python file. This approach prioritizes readability and transparency over code reuse, allowing users to see exactly how an algorithm is implemented without jumping between multiple modules.

Who it’s for

It is designed for researchers and developers who want to understand the inner workings of RL algorithms or prototype advanced features that are not supported by standard modular libraries.

Highlights

• Single-file implementation: Each algorithm variant is contained in one file for easy reading and debugging.
• Benchmarked results: Includes implementations for 7+ algorithms across 34+ games.
• Experiment tracking: Integrated with Tensorboard and Weights and Biases for logging and management.
• Cloud ready: Supports Docker and AWS Batch for scaling experiments to thousands of runs.
• Broad algorithm support: Implements PPO, DQN, C51, SAC, DDPG, TD3, PPG, and RND.