Off-Policy Actor-Critic

Abstract

This paper presents the first actor-critic algorithm for off-policy reinforcement learning. Our algorithm is online and incremental, and its per-time-step complexity scales linearly with the number of learned weights. Previous work on actor-critic algorithms is limited to the on-policy setting and does not take advantage of the recent advances in off-policy gradient temporal-difference learning. Off-policy techniques, such as Greedy-GQ, enable a target policy to be learned while following and obtaining data from another (behavior) policy. For many problems, however, actor-critic methods are more practical than action value methods (like Greedy-GQ) because they explicitly represent the policy; consequently, the policy can be stochastic and utilize a large action space. In this paper, we illustrate how to practically combine the generality and learning potential of off-policy learning with the flexibility in action selection given by actor-critic methods. We derive an incremental, linear time and space complexity algorithm that includes eligibility traces, prove convergence under assumptions similar to previous off-policy algorithms, and empirically show better or comparable performance to existing algorithms on standard reinforcement-learning benchmark problems.

Full version of the paper, appendix and errata included; Proceedings of the 2012 International Conference on Machine Learning

Cited by in corpus (27)

• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks
• A reinforcement learning approach to rare trajectory sampling
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Average-reward model-free reinforcement learning: a systematic review and literature mapping
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• Off-Policy Reinforcement Learning with Delayed Rewards
• Offline Reinforcement Learning with Soft Behavior Regularization
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
• Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm
• Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates
• Off-Policy Actor-Critic with Shared Experience Replay
• An Off-policy Policy Gradient Theorem Using Emphatic Weightings
• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
• Optimizing Sponsored Search Ranking Strategy by Deep Reinforcement Learning
• Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus
• Discrete Action On-Policy Learning with Action-Value Critic
• Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates
• Off-Policy Actor-Critic with Emphatic Weightings
• Parameter-Based Value Functions
• Deep Reinforcement Learning for Equal Risk Pricing and Hedging under Dynamic Expectile Risk Measures
• All-Action Policy Gradient Methods: A Numerical Integration Approach
• On- and Off-Policy Monotonic Policy Improvement