The Optimal Reward Baseline for Gradient-Based Reinforcement Learning

Abstract

There exist a number of reinforcement learning algorithms which learnby climbing the gradient of expected reward. Their long-runconvergence has been proved, even in partially observableenvironments with non-deterministic actions, and without the need fora system model. However, the variance of the gradient estimator hasbeen found to be a significant practical problem. Recent approacheshave discounted future rewards, introducing a bias-variance trade-offinto the gradient estimate. We incorporate a reward baseline into thelearning system, and show that it affects variance without introducingfurther bias. In particular, as we approach the zero-bias,high-variance parameterization, the optimal (or variance minimizing)constant reward baseline is equal to the long-term average expectedreward. Modified policy-gradient algorithms are presented, and anumber of experiments demonstrate their improvement over previous work.

Appears in Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI2001)

Cited by in corpus (25)

• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Learning What Data to Learn
• Real-world Video Adaptation with Reinforcement Learning
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
• Bayesian policy gradient and actor-critic algorithms
• Learning Linear Dynamical Systems with Semi-Parametric Least Squares
• You Impress Me: Dialogue Generation via Mutual Persona Perception
• An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient
• Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning
• Variance-Reduced Off-Policy Memory-Efficient Policy Search
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• Double Control Variates for Gradient Estimation in Discrete Latent Variable Models
• Evaluating a Generative Adversarial Framework for Information Retrieval
• End-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimization
• Policy Gradients Incorporating the Future
• Off-Policy Actor-Critic with Emphatic Weightings
• Coordinate-wise Control Variates for Deep Policy Gradients
• Efficient Reuse of Previous Experiences to Improve Policies in Real Environment
• Improving Gradient Estimation by Incorporating Sensor Data
• Learning Algebraic Recombination for Compositional Generalization