Consistent On-Line Off-Policy Evaluation

Abstract

The problem of on-line off-policy evaluation (OPE) has been actively studied in the last decade due to its importance both as a stand-alone problem and as a module in a policy improvement scheme. However, most Temporal Difference (TD) based solutions ignore the discrepancy between the stationary distribution of the behavior and target policies and its effect on the convergence limit when function approximation is applied. In this paper we propose the Consistent Off-Policy Temporal Difference (COP-TD($\lambda$, $\beta$)) algorithm that addresses this issue and reduces this bias at some computational expense. We show that COP-TD($\lambda$, $\beta$) can be designed to converge to the same value that would have been obtained by using on-policy TD($\lambda$) with the target policy. Subsequently, the proposed scheme leads to a related and promising heuristic we call log-COP-TD($\lambda$, $\beta$). Both algorithms have favorable empirical results to the current state of the art on-line OPE algorithms. Finally, our formulation sheds some new light on the recently proposed Emphatic TD learning.

Cited by in corpus (30)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
• Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
• Batch Stationary Distribution Estimation
• GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
• Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning
• Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
• Nearly Horizon-Free Offline Reinforcement Learning
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Generalized Off-Policy Actor-Critic
• Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation
• Variance-Reduced Off-Policy Memory-Efficient Policy Search
• Learning Retrospective Knowledge with Reverse Reinforcement Learning
• Truncated Emphatic Temporal Difference Methods for Prediction and Control
• Forethought and Hindsight in Credit Assignment
• A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning
• Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shift
• Off-Policy Actor-Critic with Emphatic Weightings
• A Unified Off-Policy Evaluation Approach for General Value Function
• Conditional Importance Sampling for Off-Policy Learning
• Infinite-horizon Off-Policy Policy Evaluation with Multiple Behavior Policies
• Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch