Rarely-switching linear bandits: optimization of causal effects for the real world
arXiv:1905.13121
Abstract
Excessively changing policies in many real world scenarios is difficult, unethical, or expensive. After all, doctor guidelines, tax codes, and price lists can only be reprinted so often. We may thus want to only change a policy when it is probable that the change is beneficial. In cases that a policy is a threshold on contextual variables we can estimate treatment effects for populations lying at the threshold. This allows for a schedule of incremental policy updates that let us optimize a policy while making few detrimental changes. Using this idea, and the theory of linear contextual bandits, we present a conservative policy updating procedure which updates a deterministic policy only when justified. We extend the theory of linear bandits to this rarely-switching case, proving that such procedures share the same regret, up to constant scaling, as the common LinUCB algorithm. However the algorithm makes far fewer changes to its policy and, of those changes, fewer are detrimental. We provide simulations and an analysis of an infant health well-being causal inference dataset, showing the algorithm efficiently learns a good policy with few changes. Our approach allows efficiently solving problems where changes are to be avoided, with potential applications in medicine, economics and beyond.
arXiv comment17 pages, 9 figures
References in corpus (13)
- A Contextual-Bandit Approach to Personalized News Article Recommendation
- Lyapunov-based Safe Policy Optimization for Continuous Control
- Estimating individual treatment effect: generalization bounds and algorithms
- Risk-Averse Multi-Armed Bandit Problems under Mean-Variance Measure
- A Survey on Contextual Multi-armed Bandits
- Causal Bandits: Learning Good Interventions via Causal Inference
- A Contextual Bandit Bake-off
- Conservative Contextual Linear Bandits
- Practical Contextual Bandits with Regression Oracles
- Generalized Risk-Aversion in Stochastic Multi-Armed Bandits
- Nonparametric Stochastic Contextual Bandits
- Learning Modular Safe Policies in the Bandit Setting with Application to Adaptive Clinical Trials
- Bandits with Movement Costs and Adaptive Pricing