Adaptive Trade-Offs in Off-Policy Learning

Abstract

A great variety of off-policy learning algorithms exist in the literature, and new breakthroughs in this area continue to be made, improving theoretical understanding and yielding state-of-the-art reinforcement learning algorithms. In this paper, we take a unifying view of this space of algorithms, and consider their trade-offs of three fundamental quantities: update variance, fixed-point bias, and contraction rate. This leads to new perspectives of existing methods, and also naturally yields novel algorithms for off-policy evaluation and control. We develop one such algorithm, C-trace, demonstrating that it is able to more efficiently make these trade-offs than existing methods in use, and that it can be scaled to yield state-of-the-art performance in large-scale environments.

AISTATS 2020 camera-ready version

References in corpus (6)

• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target
• Multi-step Off-policy Learning Without Importance Sampling Ratios
• TBQ($\sigma$): Improving Efficiency of Trace Utilization for Off-Policy Reinforcement Learning

Cited by in corpus (7)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• A Self-Tuning Actor-Critic Algorithm
• On component interactions in two-stage recommender systems
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Taylor Expansion Policy Optimization
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation