Multi-step Off-policy Learning Without Importance Sampling Ratios

Abstract

To estimate the value functions of policies from exploratory data, most model-free off-policy algorithms rely on importance sampling, where the use of importance sampling ratios often leads to estimates with severe variance. It is thus desirable to learn off-policy without using the ratios. However, such an algorithm does not exist for multi-step learning with function approximation. In this paper, we introduce the first such algorithm based on temporal-difference (TD) learning updates. We show that an explicit use of importance sampling ratios can be eliminated by varying the amount of bootstrapping in TD updates in an action-dependent manner. Our new algorithm achieves stability using a two-timescale gradient-based TD update. A prior algorithm based on lookup table representation called Tree Backup can also be retrieved using action-dependent bootstrapping, becoming a special case of our algorithm. In two challenging off-policy tasks, we demonstrate that our algorithm is stable, effectively avoids the large variance issue, and can perform substantially better than its state-of-the-art counterpart.

24 pages, 4 figures

References in corpus (2)

• Doubly Robust Policy Evaluation and Learning
• Adaptive Lambda Least-Squares Temporal Difference Learning

Cited by in corpus (4)

• A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
• Importance Sampling based Exploration in Q Learning
• An Empirical Comparison of Off-policy Prediction Learning Algorithms in the Four Rooms Environment
• An Empirical Comparison of Off-policy Prediction Learning Algorithms on the Collision Task