HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents

Abstract

Even though reinforcement-learning-based algorithms achieved superhuman performance in many domains, the field of robotics poses significant challenges as the state and action spaces are continuous, and the reward function is predominantly sparse. Furthermore, on many occasions, the agent is devoid of access to any form of demonstration. Inspired by human learning, in this work, we propose a method named highlight experience replay (HiER) that creates a secondary highlight replay buffer for the most relevant experiences. For the weights update, the transitions are sampled from both the standard and the highlight experience replay buffer. It can be applied with or without the techniques of hindsight experience replay (HER) and prioritized experience replay (PER). Our method significantly improves the performance of the state-of-the-art, validated on 8 tasks of three robotic benchmarks. Furthermore, to exploit the full potential of HiER, we propose HiER+ in which HiER is enhanced with an arbitrary data collection curriculum learning method. Our implementation, the qualitative results, and a video presentation are available on the project site: http://www.danielhorvath.eu/hier/.

Published in IEEE Access

References in corpus (19)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Self-Imitation Learning
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Learning Montezuma's Revenge from a Single Demonstration
• Object Detection Using Sim2Real Domain Randomization for Robotic Applications
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Accelerating Reinforcement Learning for Reaching using Continuous Curriculum Learning
• Measuring the Reliability of Reinforcement Learning Algorithms
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• panda-gym: Open-source goal-conditioned environments for robotic learning
• Automated curricula through setter-solver interactions
• Self-Imitation Advantage Learning
• Sim2Real Grasp Pose Estimation for Adaptive Robotic Applications