Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters

Abstract

In this work, we consider the problem of estimating a behaviour policy for use in Off-Policy Policy Evaluation (OPE) when the true behaviour policy is unknown. Via a series of empirical studies, we demonstrate how accurate OPE is strongly dependent on the calibration of estimated behaviour policy models: how precisely the behaviour policy is estimated from data. We show how powerful parametric models such as neural networks can result in highly uncalibrated behaviour policy models on a real-world medical dataset, and illustrate how a simple, non-parametric, k-nearest neighbours model produces better calibrated behaviour policy estimates and can be used to obtain superior importance sampling-based OPE estimates.

arXiv commentAccepted to workshop on Machine Learning for Causal Inference, Counterfactual Prediction, and Autonomous Action at ICML 2018

References in corpus (3)

• On Calibration of Modern Neural Networks
• Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

Cited by in corpus (8)

• Reinforcement Learning in Healthcare: A Survey
• Model-Based Reinforcement Learning for Sepsis Treatment
• Evaluating the Robustness of Off-Policy Evaluation
• Benchmarks for Deep Off-Policy Evaluation
• POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Identifying Distinct, Effective Treatments for Acute Hypotension with SODA-RL: Safely Optimized Diverse Accurate Reinforcement Learning
• Understanding the Artificial Intelligence Clinician and optimal treatment strategies for sepsis in intensive care