On Proximal Policy Optimization's Heavy-tailed Gradients
arXiv:2102.10264
Abstract
Modern policy gradient algorithms such as Proximal Policy Optimization (PPO) rely on an arsenal of heuristics, including loss clipping and gradient clipping, to ensure successful learning. These heuristics are reminiscent of techniques from robust statistics, commonly used for estimation in outlier-rich (``heavy-tailed'') regimes. In this paper, we present a detailed empirical study to characterize the heavy-tailed nature of the gradients of the PPO surrogate reward function. We demonstrate that the gradients, especially for the actor network, exhibit pronounced heavy-tailedness and that it increases as the agent's policy diverges from the behavioral policy (i.e., as the agent goes further off policy). Further examination implicates the likelihood ratios and advantages in the surrogate reward as the main sources of the observed heavy-tailedness. We then highlight issues arising due to the heavy-tailed nature of the gradients. In this light, we study the effects of the standard PPO clipping heuristics, demonstrating that these tricks primarily serve to offset heavy-tailedness in gradients. Thus motivated, we propose incorporating GMOM, a high-dimensional robust estimator, into PPO as a substitute for three clipping tricks. Despite requiring less hyperparameter tuning, our method matches the performance of PPO (with all heuristics enabled) on a battery of MuJoCo continuous control tasks.
arXiv commentICML 2021
References in corpus (14)
- Trust Region Policy Optimization
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Dota 2 with Large Scale Deep Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
- Why gradient clipping accelerates training: A theoretical justification for adaptivity
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
- A Closer Look at Deep Policy Gradients
- Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
- Variational Inference with Tail-adaptive f-Divergence
- Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
- Beyond variance reduction: Understanding the true impact of baselines on policy optimization