Generalized Off-Policy Actor-Critic
arXiv:1903.11329
Abstract
We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our new objective better predicts such performance. We prove the Generalized Off-Policy Policy Gradient Theorem to compute the policy gradient of the counterfactual objective and use an emphatic approach to get an unbiased sample from this policy gradient, yielding the Generalized Off-Policy Actor-Critic (Geoff-PAC) algorithm. We demonstrate the merits of Geoff-PAC over existing algorithms in Mujoco robot simulation tasks, the first empirical success of emphatic algorithms in prevailing deep RL benchmarks.
arXiv commentNeurIPS 2019
References in corpus (18)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Sample Efficient Actor-Critic with Experience Replay
- Equivalence Between Policy Gradients and Soft Q-Learning
- Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
- Combining policy gradient and Q-learning
- Efficient Parallel Methods for Deep Reinforcement Learning
- On Convergence of Emphatic Temporal-Difference Learning
- Consistent On-Line Off-Policy Evaluation
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
- Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation
- Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis
- Online Off-policy Prediction
- Sample-efficient Deep Reinforcement Learning for Dialog Control