Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality

Abstract

Designing off-policy reinforcement learning algorithms is typically a very challenging task, because a desirable iteration update often involves an expectation over an on-policy distribution. Prior off-policy actor-critic (AC) algorithms have introduced a new critic that uses the density ratio for adjusting the distribution mismatch in order to stabilize the convergence, but at the cost of potentially introducing high biases due to the estimation errors of both the density ratio and value function. In this paper, we develop a doubly robust off-policy AC (DR-Off-PAC) for discounted MDP, which can take advantage of learned nuisance functions to reduce estimation errors. Moreover, DR-Off-PAC adopts a single timescale structure, in which both actor and critics are updated simultaneously with constant stepsize, and is thus more sample efficient than prior algorithms that adopt either two timescale or nested-loop structure. We study the finite-time convergence rate and characterize the sample complexity for DR-Off-PAC to attain an $\epsilon$-accurate optimal policy. We also show that the overall convergence of DR-Off-PAC is doubly robust to the approximation errors that depend only on the expressive power of approximation functions. To the best of our knowledge, our study establishes the first overall sample complexity analysis for a single time-scale off-policy AC algorithm.

arXiv commentPublished in ICML 2021

References in corpus (24)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Reinforcement Learning with Deep Energy-Based Policies
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Evolved Policy Gradients
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• GenDICE: Generalized Offline Estimation of Stationary Values
• Off-Policy Policy Gradient with State Distribution Correction
• Two Time-scale Off-Policy TD Learning: Non-asymptotic Analysis over Markovian Samples
• Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods
• Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithms
• An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
• Off-Policy Evaluation via the Regularized Lagrangian
• Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation
• Reanalysis of Variance Reduced Temporal Difference Learning
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
• Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
• Variance-Reduced Off-Policy Memory-Efficient Policy Search

Cited by in corpus (4)

• Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis
• A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning
• A Unified Off-Policy Evaluation Approach for General Value Function
• Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch