OpenAI Study on Count-Based Exploration for Deep Reinforcement Learning
TL;DR
OpenAI has introduced a simple generalization of classic count-based exploration that uses hash codes to map high-dimensional states to a hash table. This approach allows count-based methods to scale to high-dimensional and continuous state spaces, achieving near state-of-the-art performance on Atari 2600 games and continuous control tasks.
Scaling Count-Based Exploration to High-Dimensional Spaces
Traditional count-based exploration algorithms perform near-optimally in small, discrete Markov decision processes (MDPs) using tabular reinforcement learning. However, they have historically been considered inapplicable to high-dimensional or continuous state spaces because most states are unlikely to occur more than once, making simple counting impossible.
To overcome this challenge, OpenAI researchers found that mapping states to hash codes allows for the efficient counting of state occurrences. By using a hash table to track these counts, the agent can compute a reward bonus based on classic count-based exploration theory, providing a reward for visiting own newly discovered states.
Performance and Benchmarks
The proposed hash-based counting method achieves near state-of-the-art performance on various deep RL benchmarks, including:
Atari 2600 games: The method provides a powerful baseline for solving MDPs that require significant exploration.
Continuous control tasks: The method scales effectively to high-dimensional continuous state spaces.
This approach serves as a simple yet powerful baseline for comparison against more complex heuristics, such as those relying on intrinsic motivation or optimism in the face of uncertainty.
Key Factors for Effective Hashing
Detailed analysis reveals that the effectiveness of the hash function used for exploration is critical. A successful hash function must possess two primary characteristics:
Appropriate Granularity: The function must map states that are sufficiently similar to the same hash code, effectively grouping similar states together.
Relevant Information Encoding: The function must encode information that is relevant to solving the specific MDP being tackled.
The researchers found that while simple hash functions can achieve surprisingly good results, a domain-dependent learned hash code can further improve performance results.