Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines

Abstract

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

References in corpus (9)

• A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patients
• Veridical Data Science
• Bandit Problems with Side Observations
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
• Educational Note: Paradoxical Collider Effect in the Analysis of Non-Communicable Disease Epidemiological Data: a reproducible illustration and web application
• Multi-Task Learning for Contextual Bandits
• Power Constrained Bandits
• Horde of Bandits using Gaussian Markov Random Fields
• Next Waves in Veridical Network Embedding