A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning

Abstract

Many reinforcement learning algorithms rely on value estimation, however, the most widely used algorithms -- namely temporal difference algorithms -- can diverge under both off-policy sampling and nonlinear function approximation. Many algorithms have been developed for off-policy value estimation based on the linear mean squared projected Bellman error (MSPBE) and are sound under linear function approximation. Extending these methods to the nonlinear case has been largely unsuccessful. Recently, several methods have been introduced that approximate a different objective -- the mean-squared Bellman error (MSBE) -- which naturally facilitate nonlinear approximation. In this work, we build on these insights and introduce a new generalized MSPBE that extends the linear MSPBE to the nonlinear setting. We show how this generalized objective unifies previous work and obtain new bounds for the value error of the solutions of the generalized objective. We derive an easy-to-use, but sound, algorithm to minimize the generalized objective, and show that it is more stable across runs, is less sensitive to hyperparameters, and performs favorably across four control domains with neural network function approximation.

Accepted for publication in JMLR 2022

References in corpus (15)

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Stochastic Variance Reduction Methods for Policy Evaluation
• Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces
• Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view
• Consistent On-Line Off-Policy Evaluation
• Convergent Tree Backup and Retrace with Function Approximation
• Multi-step Off-policy Learning Without Importance Sampling Ratios
• A Kernel Loss for Solving the Bellman Equation
• Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis
• Investigating practical linear temporal difference learning
• Online Off-policy Prediction
• Gradient Temporal-Difference Learning with Regularized Corrections
• Representations for Stable Off-Policy Reinforcement Learning