"Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment

Abstract

Artificial intelligence (AI) technology has been increasingly used in the implementation of advanced Clinical Decision Support Systems (CDSS). Research demonstrated the potential usefulness of AI-powered CDSS (AI-CDSS) in clinical decision making scenarios. However, post-adoption user perception and experience remain understudied, especially in developing countries. Through observations and interviews with 22 clinicians from 6 rural clinics in China, this paper reports the various tensions between the design of an AI-CDSS system ("Brilliant Doctor") and the rural clinical context, such as the misalignment with local context and workflow, the technical limitations and usability barriers, as well as issues related to transparency and trustworthiness of AI-CDSS. Despite these tensions, all participants expressed positive attitudes toward the future of AI-CDSS, especially acting as "a doctor's AI assistant" to realize a Human-AI Collaboration future in clinical settings. Finally we draw on our findings to discuss implications for designing AI-CDSS interventions for rural clinical contexts in developing countries.

References in corpus (7)

• Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes
• Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems
• Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
• AutoDS: Towards Human-Centered Automation of Data Science
• AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates
• CASS: Towards Building a Social-Support Chatbot for Online Health Community
• Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML

Cited by in corpus (13)

• Challenges in Deploying Machine Learning: a Survey of Case Studies
• Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis
• AutoDS: Towards Human-Centered Automation of Data Science
• `It is currently hodgepodge'': Examining AI/ML Practitioners' Challenges during Co-production of Responsible AI Values
• "If I Had All the Time in the World": Ophthalmologists' Perceptions of Anchoring Bias Mitigation in Clinical AI Support
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System
• Understanding Human-AI Collaboration in Music Therapy Through Co-Design with Therapists
• CASS: Towards Building a Social-Support Chatbot for Online Health Community
• One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images
• "It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
• Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
• Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks