Towards Robust Off-Policy Evaluation via Human Inputs

Abstract

Off-policy Evaluation (OPE) methods are crucial tools for evaluating policies in high-stakes domains such as healthcare, where direct deployment is often infeasible, unethical, or expensive. When deployment environments are expected to undergo changes (that is, dataset shifts), it is important for OPE methods to perform robust evaluation of the policies amidst such changes. Existing approaches consider robustness against a large class of shifts that can arbitrarily change any observable property of the environment. This often results in highly pessimistic estimates of the utilities, thereby invalidating policies that might have been useful in deployment. In this work, we address the aforementioned problem by investigating how domain knowledge can help provide more realistic estimates of the utilities of policies. We leverage human inputs on which aspects of the environments may plausibly change, and adapt the OPE methods to only consider shifts on these aspects. Specifically, we propose a novel framework, Robust OPE (ROPE), which considers shifts on a subset of covariates in the data based on user inputs, and estimates worst-case utility under these shifts. We then develop computationally efficient algorithms for OPE that are robust to the aforementioned shifts for contextual bandits and Markov decision processes. We also theoretically analyze the sample complexity of these algorithms. Extensive experimentation with synthetic and real world datasets from the healthcare domain demonstrates that our approach not only captures realistic dataset shifts accurately, but also results in less pessimistic policy evaluations.

arXiv comment10 pages, 5 figures, 1 table. Appeared at AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. Expanded version of arXiv:2103.15933

References in corpus (20)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Explaining and Harnessing Adversarial Examples
• Rényi Divergence and Kullback-Leibler Divergence
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Robust Adversarial Reinforcement Learning
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Doubly Robust Policy Evaluation and Optimization
• Does Distributionally Robust Supervised Learning Give Robust Classifiers?
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• Adversarial Robustness Against the Union of Multiple Perturbation Models
• Anchor regression: heterogeneous data meets causality
• Invariant Causal Prediction for Block MDPs
• Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model
• Evaluating Model Robustness and Stability to Dataset Shift
• Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs
• Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
• POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning
• Generalizability of predictive models for intensive care unit patients
• A short note on an inequality between KL and TV
• Robust Batch Policy Learning in Markov Decision Processes