Importance Resampling for Off-policy Prediction

Abstract

Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the weights for the value function. In this work, we explore a resampling strategy as an alternative to reweighting. We propose Importance Resampling (IR) for off-policy prediction, which resamples experience from a replay buffer and applies standard on-policy updates. The approach avoids using importance sampling ratios in the update, instead correcting the distribution before the update. We characterize the bias and consistency of IR, particularly compared to Weighted IS (WIS). We demonstrate in several microworlds that IR has improved sample efficiency and lower variance updates, as compared to IS and several variance-reduced IS strategies, including variants of WIS and V-trace which clips IS ratios. We also provide a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.

Recently published in NeurIPS 2019

References in corpus (8)

• End to End Learning for Self-Driving Cars
• Prioritized Experience Replay
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Train faster, generalize better: Stability of stochastic gradient descent
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Effective Sample Size for Importance Sampling based on discrepancy measures
• The Predictron: End-To-End Learning and Planning
• Multi-step Off-policy Learning Without Importance Sampling Ratios

Cited by in corpus (2)

• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Adaptive Experience Selection for Policy Gradient