A Closer Look at Deep Policy Gradients
arXiv:1811.02553
Abstract
We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a fine-grained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the "true" gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods.
arXiv commentICLR 2020 version
References in corpus (8)
- Trust Region Policy Optimization
- Don't Decay the Learning Rate, Increase the Batch Size
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Simple random search provides a competitive approach to reinforcement learning
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- An Empirical Model of Large-Batch Training
- On the Computational Efficiency of Training Neural Networks
- Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
Cited by in corpus (12)
- Queueing Network Controls via Deep Reinforcement Learning
- Investigating the effect of circuit cutting in QAOA for the MaxCut problem on NISQ devices
- Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
- How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
- Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
- A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
- Proximal Policy Optimization Smoothed Algorithm
- Learning Value Functions in Deep Policy Gradients using Residual Variance
- On Proximal Policy Optimization's Heavy-tailed Gradients
- Continuous Control With Ensemble Deep Deterministic Policy Gradients
- Recomposing the Reinforcement Learning Building Blocks with Hypernetworks
- Improved Memories Learning