Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
arXiv:1911.04384
Abstract
We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained via Gradient Emphasis Learning (GEM), a novel combination of the key ideas of Gradient Temporal Difference Learning and Emphatic Temporal Difference Learning. With the help of the emphasis critic and the canonical value function critic, we show convergence for COF-PAC, where the critics are linear and the actor can be nonlinear.
arXiv commentICML 2020
References in corpus (27)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Reinforcement Learning with Deep Energy-Based Policies
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Sample Efficient Actor-Critic with Experience Replay
- Equivalence Between Policy Gradients and Soft Q-Learning
- SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
- Finite-Sample Analysis of Proximal Gradient TD Algorithms
- Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
- Combining policy gradient and Q-learning
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Stochastic Variance Reduction Methods for Policy Evaluation
- Finite-Sample Analysis for SARSA with Linear Function Approximation
- On Convergence of Emphatic Temporal-Difference Learning
- GenDICE: Generalized Offline Estimation of Stationary Values
- Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces
- Off-Policy Policy Gradient with State Distribution Correction
- Consistent On-Line Off-Policy Evaluation
- Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithms
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
- Minimax Weight and Q-Function Learning for Off-Policy Evaluation
- Improving Sample Complexity Bounds for (Natural) Actor-Critic Algorithms
- On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy Learning
- Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation