Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens

Abstract

Major depressive disorder is a debilitating disease affecting 264 million people worldwide. While many antidepressant medications are available, few clinical guidelines support choosing among them. Decision support tools (DSTs) embodying machine learning models may help improve the treatment selection process, but often fail in clinical practice due to poor system integration. We use an iterative, co-design process to investigate clinicians' perceptions of using DSTs in antidepressant treatment decisions. We identify ways in which DSTs need to engage with the healthcare sociotechnical system, including clinical processes, patient preferences, resource constraints, and domain knowledge. Our results suggest that clinical DSTs should be designed as multi-user systems that support patient-provider collaboration and offer on-demand explanations that address discrepancies between predictions and current standards of care. Through this work, we demonstrate how current trends in explainable AI may be inappropriate for clinical environments and consider paths towards designing these tools for real-world medical systems.

To appear in ACM CHI Conference on Human Factors in Computing Systems (CHI '21), May 8-13, 2021, Yokohama, Japan

References in corpus (4)

• Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes
• CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysis

Cited by in corpus (14)

• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis
• Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
• Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health Support
• Twenty-Four Years of Empirical Research on Trust in AI: A Bibliometric Review of Trends, Overlooked Issues, and Future Directions
• Seamful XAI: Operationalizing Seamful Design in Explainable AI
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
• AI, Help Me Think$\unicode{x2014}$but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support
• Using HCI in Cross-Disciplinary Teams: A Case Study of Academic Collaboration in HCI-Health Teams in the US Using a Team Science Perspective
• Investigating Why Clinicians Deviate from Standards of Care: Liberating Patients from Mechanical Ventilation in the ICU
• Beyond Recommendations: From Backward to Forward AI Support of Pilots' Decision-Making Process
• "It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
• AI That Helps Us Help Each Other: A Proactive System for Scaffolding Mentor-Novice Collaboration in Entrepreneurship Coaching