Balanced Policy Evaluation and Learning

Abstract

We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electronic health records and in internet advertising. Existing approaches use inverse propensity weighting (or, doubly robust versions) to make historical outcome (or, residual) data look like it were generated by a new policy being evaluated or learned. But this relies on a plug-in approach that rejects data points with a decision that disagrees with the new policy, leading to high variance estimates and ineffective learning. We propose a new, balance-based approach that too makes the data look like the new policy but does so directly by finding weights that optimize for balance between the weighted data and the target policy in the given, finite sample, which is equivalent to minimizing worst-case or posterior conditional mean square error. Our policy learner proceeds as a two-level optimization problem over policies and weights. We demonstrate that this approach markedly outperforms existing ones both in evaluation and learning, which is unsurprising given the wider support of balance-based weights. We establish extensive theoretical consistency guarantees and regret bounds that support this empirical success.

Cited by in corpus (31)

• Fairness in Machine Learning: A Survey
• Double Machine Learning based Program Evaluation under Unconfoundedness
• Confounding-Robust Policy Improvement
• Doubly robust off-policy evaluation with shrinkage
• Post-Contextual-Bandit Inference
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Instrumental Variable Regression via Kernel Maximum Moment Loss
• Fair Decisions Despite Imperfect Predictions
• Learning When-to-Treat Policies
• Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders
• Policy Evaluation with Latent Confounders via Optimal Balance
• Efficient Policy Learning from Surrogate-Loss Classification Reductions
• Sparsity Double Robust Inference of Average Treatment Effects
• Nonparametric Heterogeneous Treatment Effect Estimation in Repeated Cross Sectional Designs
• Combining Offline Causal Inference and Online Bandit Learning for Data Driven Decision
• Counterfactual Evaluation of Treatment Assignment Functions with Networked Observational Data
• Statistically Efficient Off-Policy Policy Gradients
• A semiparametric instrumental variable approach to optimal treatment regimes under endogeneity
• Robust Batch Policy Learning in Markov Decision Processes
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap
• Optimal Off-Policy Evaluation from Multiple Logging Policies
• Double Robust Representation Learning for Counterfactual Prediction
• Beyond traditional assumptions in fair machine learning
• Optimal Policies for the Homogeneous Selective Labels Problem
• Generalizing Off-Policy Learning under Sample Selection Bias
• DTR Bandit: Learning to Make Response-Adaptive Decisions With Low Regret
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model
• Offline Contextual Bandits with Overparameterized Models
• Balanced Policy Evaluation and Learning for Right Censored Data
• Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networks
• Efficient Balanced Treatment Assignments for Experimentation