Switching Isotropic and Directional Exploration with Parameter Space Noise in Deep Reinforcement Learning

Abstract

This paper proposes an exploration method for deep reinforcement learning based on parameter space noise. Recent studies have experimentally shown that parameter space noise results in better exploration than the commonly used action space noise. Previous methods devised a way to update the diagonal covariance matrix of a noise distribution and did not consider the direction of the noise vector and its correlation. In addition, fast updates of the noise distribution are required to facilitate policy learning. We propose a method that deforms the noise distribution according to the accumulated returns and the noises that have led to the returns. Moreover, this method switches isotropic exploration and directional exploration in parameter space with regard to obtained rewards. We validate our exploration strategy in the OpenAI Gym continuous environments and modified environments with sparse rewards. The proposed method achieves results that are competitive with a previous method at baseline tasks. Moreover, our approach exhibits better performance in sparse reward environments by exploration with the switching strategy.

arXiv comment16 pages, 5 figures

References in corpus (11)

• Continuous control with deep reinforcement learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Noisy Networks for Exploration
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Parameter Space Noise for Exploration
• Unifying Count-Based Exploration and Intrinsic Motivation
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• On the Relationship Between the OpenAI Evolution Strategy and Stochastic Gradient Descent