Self-Imitation Advantage Learning
arXiv:2012.11989
Abstract
Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems. It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks. Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward. We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning. Crucially, our method mitigates the problem of stale returns by choosing the most optimistic return estimate between the observed return and the current action-value for self-imitation. We demonstrate the empirical effectiveness of SAIL on the Arcade Learning Environment, with a focus on hard exploration games.
AAMAS 2021
References in corpus (20)
- Playing Atari with Deep Reinforcement Learning
- Prioritized Experience Replay
- Dueling Network Architectures for Deep Reinforcement Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Hindsight Experience Replay
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Challenges of Real-World Reinforcement Learning
- Agent57: Outperforming the Atari Human Benchmark
- Go-Explore: a New Approach for Hard-Exploration Problems
- Dopamine: A Research Framework for Deep Reinforcement Learning
- Never Give Up: Learning Directed Exploration Strategies
- Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
- Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
- Generative Adversarial Self-Imitation Learning
- Munchausen Reinforcement Learning
- Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
- Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards
- Fast deep reinforcement learning using online adjustments from the past
- Self-Imitation Learning via Generalized Lower Bound Q-learning