mohmdelsayed/streaming-drl

What it solves

This project addresses the "stream barrier" in deep reinforcement learning (RL). Most deep RL algorithms rely on experience replay buffers to store past interactions, which is impractical for resource-constrained or privacy-sensitive applications. Previous attempts to implement streaming RL (learning directly from experience as it arrives) often failed to learn effectively, performing close to random policies.

How it works

The project implements the "Stream-X" recipe, a set of techniques designed to make streaming deep RL stable and effective. It combines three core components:

1. Signal normalization: Standardizing the data inputs.
2. Representation stabilization: Ensuring the neural network's internal representations remain consistent.
3. Controlled parameter updates: Managing how the model weights are adjusted during learning.

By applying this recipe to base algorithms like Stream-AC, Stream-RAC, Stream-Q, and Stream-SARSA, the system can learn from a stream of experience without needing a replay buffer.

Who it’s for

Researchers and developers working in reinforcement learning who need to train agents in environments where storing large amounts of past data is prohibited by memory limits or privacy requirements.

Highlights

• Overcomes the Stream Barrier: Provides the first family of deep RL algorithms that can learn effectively from streaming data.
• Hedge against Buffer Dependency: Reduces the reliance on batch RL algorithms like PPO, SAC, and DQN.
• Broad Compatibility: Supports a wide range of environments including Atari, MuJoCo, Classical Control, and dm_control.
• Consistent Performance: Uses a single prescribed hyperparameter configuration per algorithm across various tasks.