Reinforcement Learning in Healthcare: A Survey

Abstract

As a subfield of machine learning, reinforcement learning (RL) aims at empowering one's capabilities in behavioural decision making by using interaction experience with the world and an evaluative feedback. Unlike traditional supervised learning methods that usually rely on one-shot, exhaustive and supervised reward signals, RL tackles with sequential decision making problems with sampled, evaluative and delayed feedback simultaneously. Such distinctive features make RL technique a suitable candidate for developing powerful solutions in a variety of healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged and sequential procedure. This survey discusses the broad applications of RL techniques in healthcare domains, in order to provide the research community with systematic understanding of theoretical foundations, enabling methods and techniques, existing challenges, and new insights of this emerging paradigm. By first briefly examining theoretical foundations and key techniques in RL research from efficient and representational directions, we then provide an overview of RL applications in healthcare domains ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis from both unstructured and structured clinical data, as well as many other control or scheduling domains that have infiltrated many aspects of a healthcare system. Finally, we summarize the challenges and open issues in current research, and point out some potential solutions and directions for future research.

References in corpus (38)

• Distilling the Knowledge in a Neural Network
• Continuous control with deep reinforcement learning
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Applications of Deep Learning and Reinforcement Learning to Biological Data
• Learning to Diagnose with LSTM Recurrent Neural Networks
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patients
• Verifiable Reinforcement Learning via Policy Extraction
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
• Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• A dual mode adaptive basal-bolus advisor based on reinforcement learning
• Safe Exploration in Markov Decision Processes
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Small Sample Learning in Big Data Era
• The Doctor Just Won't Accept That!
• Deep Reinforcement Learning for Sepsis Treatment
• Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients
• Near-optimal Reinforcement Learning in Factored MDPs
• Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning
• Model-Based Reinforcement Learning for Sepsis Treatment
• Robust Hybrid Learning for Estimating Personalized Dynamic Treatment Regimens
• Agent-Agnostic Human-in-the-Loop Reinforcement Learning
• Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
• The Actor Search Tree Critic (ASTC) for Off-Policy POMDP Learning in Medical Decision Making
• A Batch, Off-Policy, Actor-Critic Algorithm for Optimizing the Average Reward
• Precision medicine as a control problem: Using simulation and deep reinforcement learning to discover adaptive, personalized multi-cytokine therapy for sepsis
• Interactive Q-learning for Probabilities and Quantiles
• A Function Approximation Method for Model-based High-Dimensional Inverse Reinforcement Learning
• Improving Mild Cognitive Impairment Prediction via Reinforcement Learning and Dialogue Simulation
• Personalized Medical Treatments Using Novel Reinforcement Learning Algorithms
• Transfer Learning Across Patient Variations with Hidden Parameter Markov Decision Processes
• An Optimal Policy for Patient Laboratory Tests in Intensive Care Units
• Towards Mixed Optimization for Reinforcement Learning with Program Synthesis
• Dynamic Measurement Scheduling for Adverse Event Forecasting using Deep RL

Cited by in corpus (28)

• On the Opportunities and Risks of Foundation Models
• Exploration in Deep Reinforcement Learning: A Survey
• Transfer Learning in Deep Reinforcement Learning: A Survey
• A Theoretical Analysis of Deep Q-Learning
• Optimizing Warfarin Dosing using Deep Reinforcement Learning
• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare
• Multistep Automated Data Labelling Procedure (MADLaP) for Thyroid Nodules on Ultrasound: An Artificial Intelligence Approach for Automating Image Annotation
• Robust Policies For Proactive ICU Transfers
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learning
• A Provably-Efficient Model-Free Algorithm for Constrained Markov Decision Processes
• Interactive Visualization for Debugging RL
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• Missingness as Stability: Understanding the Structure of Missingness in Longitudinal EHR data and its Impact on Reinforcement Learning in Healthcare
• Embedded methods for feature selection in neural networks
• Learning medical triage from clinicians using Deep Q-Learning
• Evaluating the Safety of Deep Reinforcement Learning Models using Semi-Formal Verification
• Learning to Sample with Local and Global Contexts in Experience Replay Buffer
• CaiRL: A High-Performance Reinforcement Learning Environment Toolkit
• Identifying Decision Points for Safe and Interpretable Reinforcement Learning in Hypotension Treatment
• Can Q-Learning be Improved with Advice?
• BF++: a language for general-purpose program synthesis