AlgaeDICE: Policy Gradient from Arbitrary Experience

Abstract

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.

References in corpus (5)

• Rainbow: Combining Improvements in Deep Reinforcement Learning
• A unified view of entropy-regularized Markov decision processes
• GenDICE: Generalized Offline Estimation of Stationary Values
• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• Learning from Conditional Distributions via Dual Embeddings

Cited by in corpus (2)

• Reinforcement Learning via Fenchel-Rockafellar Duality
• Statistically Efficient Off-Policy Policy Gradients