The Optimal Reward Baseline for Gradient-Based Reinforcement Learning

Abstract

There exist a number of reinforcement learning algorithms which learnby climbing the gradient of expected reward. Their long-runconvergence has been proved, even in partially observableenvironments with non-deterministic actions, and without the need fora system model. However, the variance of the gradient estimator hasbeen found to be a significant practical problem. Recent approacheshave discounted future rewards, introducing a bias-variance trade-offinto the gradient estimate. We incorporate a reward baseline into thelearning system, and show that it affects variance without introducingfurther bias. In particular, as we approach the zero-bias,high-variance parameterization, the optimal (or variance minimizing)constant reward baseline is equal to the long-term average expectedreward. Modified policy-gradient algorithms are presented, and anumber of experiments demonstrate their improvement over previous work.

Appears in Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI2001)

Cited by in corpus (60)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Neural Variational Inference and Learning in Belief Networks
• Game-Theoretic Multiagent Reinforcement Learning
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Monte Carlo Gradient Estimation in Machine Learning
• Adversarial Neural Machine Translation
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Learning What Data to Learn
• Learning Scheduling Algorithms for Data Processing Clusters
• Real-world Video Adaptation with Reinforcement Learning
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
• Bayesian policy gradient and actor-critic algorithms
• MuProp: Unbiased Backpropagation for Stochastic Neural Networks
• Compositional Generalization by Learning Analytical Expressions
• Learning Linear Dynamical Systems with Semi-Parametric Least Squares
• Efficient Gradient Estimation for Motor Control Learning
• Settling the Variance of Multi-Agent Policy Gradients
• Learning Wake-Sleep Recurrent Attention Models
• You Impress Me: Dialogue Generation via Mutual Persona Perception
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• Average-reward model-free reinforcement learning: a systematic review and literature mapping
• Multi-Vehicle Routing Problems with Soft Time Windows: A Multi-Agent Reinforcement Learning Approach
• A Study of Reinforcement Learning for Neural Machine Translation
• An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient
• Prospects of reinforcement learning for the simultaneous damping of many mechanical modes
• Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
• Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
• DiCE: The Infinitely Differentiable Monte-Carlo Estimator
• History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms
• Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Evolving to learn: discovering interpretable plasticity rules for spiking networks
• Source-Critical Reinforcement Learning for Transferring Spoken Language Understanding to a New Language
• The Thermodynamic Variational Objective
• Select and Attend: Towards Controllable Content Selection in Text Generation
• Variance-Reduced Off-Policy Memory-Efficient Policy Search
• Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning
• An Off-policy Policy Gradient Theorem Using Emphatic Weightings
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks
• Learning Value Functions in Deep Policy Gradients using Residual Variance
• Domain Transfer in Dialogue Systems without Turn-Level Supervision
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Differentiable Antithetic Sampling for Variance Reduction in Stochastic Variational Inference
• Evaluating a Generative Adversarial Framework for Information Retrieval
• End-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimization
• Policy Gradients Incorporating the Future
• Off-Policy Actor-Critic with Emphatic Weightings
• Double Control Variates for Gradient Estimation in Discrete Latent Variable Models
• A Split-and-Recombine Approach for Follow-up Query Analysis
• The Virtual Patch Clamp: Imputing C. elegans Membrane Potentials from Calcium Imaging
• A Gauss-Newton Method for Markov Decision Processes
• Improving Gradient Estimation by Incorporating Sensor Data
• Learning Algebraic Recombination for Compositional Generalization
• Coordinate-wise Control Variates for Deep Policy Gradients
• Efficient Reuse of Previous Experiences to Improve Policies in Real Environment
• An Approximate Newton Method for Markov Decision Processes