PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
arXiv:2004.11667 · doi:10.1007/978-3-030-61616-8_24
Abstract
The exploration-exploitation trade-off is at the heart of reinforcement learning (RL). However, most continuous control benchmarks used in recent RL research only require local exploration. This led to the development of algorithms that have basic exploration capabilities, and behave poorly in benchmarks that require more versatile exploration. For instance, as demonstrated in our empirical study, state-of-the-art RL algorithms such as DDPG and TD3 are unable to steer a point mass in even small 2D mazes. In this paper, we propose a new algorithm called "Plan, Backplay, Chain Skills" (PBCS) that combines motion planning and reinforcement learning to solve hard exploration environments. In a first phase, a motion planning algorithm is used to find a single good trajectory, then an RL algorithm is trained using a curriculum derived from the trajectory, by combining a variant of the Backplay algorithm and skill chaining. We show that this method outperforms state-of-the-art RL algorithms in 2D maze environments of various sizes, and is able to improve on the trajectory obtained by the motion planning phase.
References in corpus (16)
- DeepMind Control Suite
- Off-Policy Deep Reinforcement Learning without Exploration
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- Exploration by Random Network Distillation
- Go-Explore: a New Approach for Hard-Exploration Problems
- Deep Reinforcement Learning and the Deadly Triad
- Diversity is All You Need: Learning Skills without a Reward Function
- Learning Montezuma's Revenge from a Single Demonstration
- Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
- GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
- Towards Characterizing Divergence in Deep Q-Learning
- Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
- Backplay: "Man muss immer umkehren"
- Recall Traces: Backtracking Models for Efficient Reinforcement Learning
- Reinforcement Learning with Probabilistically Complete Exploration
- Temporal Difference Learning with Neural Networks - Study of the Leakage Propagation Problem