Policy Evaluation and Optimization with Continuous Treatments

Abstract

We study the problem of policy evaluation and learning from batched contextual bandit data when treatments are continuous, going beyond previous work on discrete treatments. Previous work for discrete treatment/action spaces focuses on inverse probability weighting (IPW) and doubly robust (DR) methods that use a rejection sampling approach for evaluation and the equivalent weighted classification problem for learning. In the continuous setting, this reduction fails as we would almost surely reject all observations. To tackle the case of continuous treatments, we extend the IPW and DR approaches to the continuous setting using a kernel function that leverages treatment proximity to attenuate discrete rejection. Our policy estimator is consistent and we characterize the optimal bandwidth. The resulting continuous policy optimizer (CPO) approach using our estimator achieves convergent regret and approaches the best-in-class policy for learnable policy classes. We demonstrate that the estimator performs well and, in particular, outperforms a discretization-based benchmark. We further study the performance of our policy optimizer in a case study on personalized dosing based on a dataset of Warfarin patients, their covariates, and final therapeutic doses. Our learned policy outperforms benchmarks and nears the oracle-best linear policy.

arXiv commentappearing at AISTATS 2018

Cited by in corpus (18)

• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Estimating the Effects of Continuous-valued Interventions using Generative Adversarial Networks
• Confounding-Robust Policy Improvement
• Optimization over Continuous and Multi-dimensional Decisions with Observational Data
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
• Kernel Optimal Orthogonality Weighting: A Balancing Approach to Estimating Effects of Continuous Treatments
• Adaptive Estimator Selection for Off-Policy Evaluation
• Fairness, Welfare, and Equity in Personalized Pricing
• More robust estimation of sample average treatment effects using Kernel Optimal Matching in an observational study of spine surgical interventions
• Policy Evaluation with Latent Confounders via Optimal Balance
• Characterization of Overlap in Observational Studies
• Efficient Policy Learning from Surrogate-Loss Classification Reductions
• Efficient Contextual Bandits with Continuous Actions
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
• Semi-Parametric Efficient Policy Learning with Continuous Actions
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Automatic Double Machine Learning for Continuous Treatment Effects
• Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning