The Mirage of Action-Dependent Baselines in Reinforcement Learning

Abstract

Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance. Several recent papers extend the baseline to depend on both the state and action and suggest that this significantly reduces variance and improves sample efficiency without introducing bias into the gradient estimates. To better understand this development, we decompose the variance of the policy gradient estimator and numerically show that learned state-action-dependent baselines do not in fact reduce variance over a state-dependent baseline in commonly tested benchmark domains. We confirm this unexpected result by reviewing the open-source code accompanying these prior papers, and show that subtle implementation decisions cause deviations from the methods presented in the papers and explain the source of the previously observed empirical gains. Furthermore, the variance decomposition highlights areas for improvement, which we demonstrate by illustrating a simple change to the typical value function parameterization that can significantly improve performance.

arXiv commentUpdated to ICML final submission

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Learning Continuous Control Policies by Stochastic Value Gradients
• Continuous Deep Q-Learning with Model-based Acceleration
• Doubly Robust Policy Evaluation and Learning
• Sample Efficient Actor-Critic with Experience Replay
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Combining policy gradient and Q-learning
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselines
• Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
• Time Limits in Reinforcement Learning

Cited by in corpus (25)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• SNAS: Stochastic Neural Architecture Search
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Understanding the impact of entropy on policy optimization
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• What Matters for Adversarial Imitation Learning?
• An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient
• Remember and Forget for Experience Replay
• How to Make Deep RL Work in Practice
• Reinforcement Learning for Uplift Modeling
• From Importance Sampling to Doubly Robust Policy Gradient
• P3O: Policy-on Policy-off Policy Optimization
• Variance-Reduced Off-Policy Memory-Efficient Policy Search
• Reward-estimation variance elimination in sequential decision processes
• Learning Value Functions in Deep Policy Gradients using Residual Variance
• Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Policy Optimization with Second-Order Advantage Information
• Which Channel to Ask My Question? Personalized Customer Service Request Stream Routing using Deep Reinforcement Learning
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Search in Imperfect Information Games
• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning
• Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy