Competitive Experience Replay
arXiv:1902.00528
Abstract
Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required. It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals. We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents. Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum. We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm. Each task provides only binary rewards indicating whether or not the goal is achieved. Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration. Extensive experiments demonstrate that this method leads to faster converge and improved task performance.
Published as a conference paper at Seventh International Conference on Learning Representations(ICLR 2019)
References in corpus (27)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Trust Region Policy Optimization
- Prioritized Experience Replay
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Emergence of Locomotion Behaviours in Rich Environments
- Distributed Prioritized Experience Replay
- Parameter Space Noise for Exploration
- Large-Scale Study of Curiosity-Driven Learning
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Continuous Deep Q-Learning with Model-based Acceleration
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Learning Dexterous In-Hand Manipulation
- Sample Efficient Actor-Critic with Experience Replay
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
- Automatic Goal Generation for Reinforcement Learning Agents
- Reverse Curriculum Generation for Reinforcement Learning
- Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
- EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
- Automated Curriculum Learning for Neural Networks
- Recall Traces: Backtracking Models for Efficient Reinforcement Learning
- Backplay: "Man muss immer umkehren"
- Forward-Backward Reinforcement Learning