Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity

Abstract

With the recent evolution of mobile health technologies, health scientists are increasingly interested in developing just-in-time adaptive interventions (JITAIs), typically delivered via notification on mobile device and designed to help the user prevent negative health outcomes and promote the adoption and maintenance of healthy behaviors. A JITAI involves a sequence of decision rules (i.e., treatment policy) that takes the user's current context as input and specifies whether and what type of an intervention should be provided at the moment. In this paper, we develop a Reinforcement Learning (RL) algorithm that continuously learns and improves the treatment policy embedded in the JITAI as the data is being collected from the user. This work is motivated by our collaboration on designing the RL algorithm in HeartSteps V2 based on data from HeartSteps V1. HeartSteps is a physical activity mobile health application. The RL algorithm developed in this paper is being used in HeartSteps V2 to decide, five times per day, whether to deliver a context-tailored activity suggestion.

References in corpus (11)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Thompson Sampling for Contextual Bandits with Linear Payoffs
• (More) Efficient Reinforcement Learning via Posterior Sampling
• A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patients
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
• Why is Posterior Sampling Better than Optimism for Reinforcement Learning?
• Estimation Considerations in Contextual Bandits
• Non-Stationary Bandits with Habituation and Recovery Dynamics
• On Optimistic versus Randomized Exploration in Reinforcement Learning
• Mitigating Planner Overfitting in Model-Based Reinforcement Learning

Cited by in corpus (4)

• Recommender systems and reinforcement learning for human-building interaction and context-aware support: A text mining-driven review of scientific literature
• Online learning in bandits with predicted context
• Thompson sampling for zero-inflated count outcomes with an application to the Drink Less mobile health study
• Effect-Invariant Mechanisms for Policy Generalization