Off-Policy Policy Gradient with State Distribution Correction

Abstract

We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what would be the distribution of states under the learned policy. Here we build on recent progress for estimating the ratio of the state distributions under behavior and evaluation policies for policy evaluation, and present an off-policy policy gradient optimization technique that can account for this mismatch in distributions. We present an illustrative example of why this is important and a theoretical convergence guarantee for our approach. Empirically, we compare our method in simulations to several strong baselines which do not correct for this mismatch, significantly improving in the quality of the policy discovered.

to appear at UAI 18; camera-ready version

References in corpus (8)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Sample Efficient Actor-Critic with Experience Replay
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Policy Optimization via Importance Sampling
• Generalized Off-Policy Actor-Critic

Cited by in corpus (27)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• A Survey of Deep Learning for Scientific Discovery
• COMBO: Conservative Offline Model-Based Policy Optimization
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Batch Value-function Approximation with Only Realizability
• Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
• Offline Reinforcement Learning from Images with Latent Space Models
• Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• Nearly Horizon-Free Offline Reinforcement Learning
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity
• Sublinear Optimal Policy Value Estimation in Contextual Bandits
• Statistically Efficient Off-Policy Policy Gradients
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Parameter-Based Value Functions
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shift
• Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch