AlgaeDICE: Policy Gradient from Arbitrary Experience

Abstract

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.

References in corpus (5)

• Rainbow: Combining Improvements in Deep Reinforcement Learning
• A unified view of entropy-regularized Markov decision processes
• GenDICE: Generalized Offline Estimation of Stationary Values
• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• Learning from Conditional Distributions via Dual Embeddings

Cited by in corpus (15)

• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Off-Policy Evaluation via the Regularized Lagrangian
• Offline Reinforcement Learning from Images with Latent Space Models
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Offline Reinforcement Learning with Soft Behavior Regularization
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Statistically Efficient Off-Policy Policy Gradients
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Near Optimal Policy Optimization via REPS
• Policy Gradients Incorporating the Future
• Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds
• A maximum-entropy approach to off-policy evaluation in average-reward MDPs
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport