Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges

Abstract

Multi-armed bandit problems (MABPs) are a special type of optimal control problem well suited to model resource allocation under uncertainty in a wide variety of contexts. Since the first publication of the optimal solution of the classic MABP by a dynamic index rule, the bandit literature quickly diversified and emerged as an active research topic. Across this literature, the use of bandit models to optimally design clinical trials became a typical motivating application, yet little of the resulting theory has ever been used in the actual design and analysis of clinical trials. To this end, we review two MABP decision-theoretic approaches to the optimal allocation of treatments in a clinical trial: the infinite-horizon Bayesian Bernoulli MABP and the finite-horizon variant. These models possess distinct theoretical properties and lead to separate allocation rules in a clinical trial design context. We evaluate their performance compared to other allocation rules, including fixed randomization. Our results indicate that bandit approaches offer significant advantages, in terms of assigning more patients to better treatments, and severe limitations, in terms of their resulting statistical power. We propose a novel bandit-based patient allocation rule that overcomes the issue of low power, thus removing a potential barrier for their use in practice.

arXiv commentPublished at http://dx.doi.org/10.1214/14-STS504 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (3)

• Computing a classic index for finite-horizon bandits
• A $(2/3)n^3$ fast-pivoting algorithm for the Gittins index and optimal stopping of a Markov chain
• Celebrating 70: An Interview with Don Berry

Cited by in corpus (12)

• Resilient Composition of Drone Services for Delivery
• Whittle index based Q-learning for restless bandits with average reward
• Multi-Armed Bandits with Correlated Arms
• Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions
• Testing Indexability and Computing Whittle and Gittins Index in Subcubic Time
• Social decision-making driven by artistic explore-exploit tension
• Best-Arm Identification in Correlated Multi-Armed Bandits
• Autonomous Drug Design with Multi-Armed Bandits
• Some performance considerations when using multi-armed bandit algorithms in the presence of missing data
• Implementing Response-Adaptive Randomisation in Stratified Rare-disease Trials: Design Challenges and Practical Solutions
• IRL for Restless Multi-Armed Bandits with Applications in Maternal and Child Health
• Understanding the stochastic dynamics of sequential decision-making processes: A path-integral analysis of multi-armed bandits