Understanding the impact of entropy on policy optimization

Abstract

Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with \emph{exploration} by encouraging the selection of more stochastic policies. In this work, we analyze this claim using new visualizations of the optimization landscape based on randomly perturbing the loss function. We first show that even with access to the exact gradient, policy optimization is difficult due to the geometry of the objective function. Then, we qualitatively show that in some environments, a policy with higher entropy can make the optimization landscape smoother, thereby connecting local optima and enabling the use of larger learning rates. This paper presents new tools for understanding the optimization landscape, shows that policy entropy serves as a regularizer, and highlights the challenge of designing general-purpose policy optimization algorithms.

arXiv commentAccepted to ICML 2019

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• How Does Batch Normalization Help Optimization?
• Qualitatively characterizing neural network optimization problems
• Equivalence Between Policy Gradients and Soft Q-Learning
• Essentially No Barriers in Neural Network Energy Landscape
• A unified view of entropy-regularized Markov decision processes
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• Clipped Action Policy Gradient
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods

Cited by in corpus (10)

• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• If MaxEnt RL is the Answer, What is the Question?
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• Observational Overfitting in Reinforcement Learning
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• An Empirical Study on Hyperparameters and their Interdependence for RL Generalization
• Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning
• Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods
• Learning Vehicle Routing Problems using Policy Optimisation