Multi-Path Policy Optimization
arXiv:1911.04207
Abstract
Recent years have witnessed a tremendous improvement of deep reinforcement learning. However, a challenging problem is that an agent may suffer from inefficient exploration, particularly for on-policy methods. Previous exploration methods either rely on complex structure to estimate the novelty of states, or incur sensitive hyper-parameters causing instability. We propose an efficient exploration method, Multi-Path Policy Optimization (MPPO), which does not incur high computation cost and ensures stability. MPPO maintains an efficient mechanism that effectively utilizes a population of diverse policies to enable better exploration, especially in sparse environments. We also give a theoretical guarantee of the stable performance. We build our scheme upon two widely-adopted on-policy methods, the Trust-Region Policy Optimization algorithm and Proximal Policy Optimization algorithm. We conduct extensive experiments on several MuJoCo tasks and their sparsified variants to fairly evaluate the proposed method. Results show that MPPO significantly outperforms state-of-the-art exploration methods in terms of both sample efficiency and final performance.
AAMAS-2020
References in corpus (12)
- Trust Region Policy Optimization
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
- Reinforcement Learning with Deep Energy-Based Policies
- Deep Reinforcement Learning that Matters
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
- Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
- Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
- Evolution-Guided Policy Gradient in Reinforcement Learning
- EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
- Policy Optimization by Genetic Distillation