AlgaeDICE: Policy Gradient from Arbitrary Experience

Abstract

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.

References in corpus (20)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• A unified view of entropy-regularized Markov decision processes
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• GenDICE: Generalized Offline Estimation of Stationary Values
• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• Scalable Bilinear $\pi$ Learning Using State and Action Features
• Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real
• A Kernel Loss for Solving the Bellman Equation
• Learning from Conditional Distributions via Dual Embeddings
• Smoothed Action Value Functions for Learning Gaussian Policies
• Stochastic Primal-Dual Q-Learning
• f-Divergence constrained policy improvement

Cited by in corpus (48)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• A Minimalist Approach to Offline Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• COMBO: Conservative Offline Model-Based Policy Optimization
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Is Pessimism Provably Efficient for Offline RL?
• Off-Policy Evaluation via the Regularized Lagrangian
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning
• CoinDICE: Off-Policy Confidence Interval Estimation
• Offline RL Without Off-Policy Evaluation
• Offline Reinforcement Learning from Images with Latent Space Models
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Offline Reinforcement Learning with Soft Behavior Regularization
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Stable Policy Optimization via Off-Policy Divergence Regularization
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Nearly Horizon-Free Offline Reinforcement Learning
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• Fast Rates for the Regret of Offline Reinforcement Learning
• Statistically Efficient Off-Policy Policy Gradients
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Bounds
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Near Optimal Policy Optimization via REPS
• Policy Gradients Incorporating the Future
• Parameter-Based Value Functions
• RAIL: A modular framework for Reinforcement-learning-based Adversarial Imitation Learning
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport
• A maximum-entropy approach to off-policy evaluation in average-reward MDPs
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach