Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods

Abstract

Recent analyses of certain gradient descent optimization methods have shown that performance can degrade in some settings - such as with stochasticity or implicit momentum. In deep reinforcement learning (Deep RL), such optimization methods are often used for training neural networks via the temporal difference error or policy gradient. As an agent improves over time, the optimization target changes and thus the loss landscape (and local optima) change. Due to the failure modes of those methods, the ideal choice of optimizer for Deep RL remains unclear. As such, we provide an empirical analysis of the effects that a wide range of gradient descent optimizers and their hyperparameters have on policy gradient methods, a subset of Deep RL algorithms, for benchmark continuous control tasks. We find that adaptive optimizers have a narrow window of effective learning rates, diverging in other cases, and that the effectiveness of momentum varies depending on the properties of the environment. Our analysis suggests that there is significant interplay between the dynamics of the environment and Deep RL algorithm properties which aren't necessarily accounted for by traditional adaptive gradient methods. We provide suggestions for optimal settings of current methods and further lines of research based on our findings.

arXiv commentAccepted at the European Workshop on Reinforcement Learning 2018 (EWRL14)

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• An overview of gradient descent optimization algorithms
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Emergence of Locomotion Behaviours in Rich Environments
• Distributed Distributional Deterministic Policy Gradients
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• YellowFin and the Art of Momentum Tuning
• Distributed Evaluations: Ending Neural Point Metrics

Cited by in corpus (9)

• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• Understanding the impact of entropy on policy optimization
• A Closer Look at Deep Policy Gradients
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• On Proximal Policy Optimization's Heavy-tailed Gradients
• TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?
• Towards Combining On-Off-Policy Methods for Real-World Applications
• Parameter-free Gradient Temporal Difference Learning