Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes

Abstract

Clinical decision support tools (DST) promise improved healthcare outcomes by offering data-driven insights. While effective in lab settings, almost all DSTs have failed in practice. Empirical research diagnosed poor contextual fit as the cause. This paper describes the design and field evaluation of a radically new form of DST. It automatically generates slides for clinicians' decision meetings with subtly embedded machine prognostics. This design took inspiration from the notion of "Unremarkable Computing", that by augmenting the users' routines technology/AI can have significant importance for the users yet remain unobtrusive. Our field evaluation suggests clinicians are more likely to encounter and embrace such a DST. Drawing on their responses, we discuss the importance and intricacies of finding the right level of unremarkableness in DST design, and share lessons learned in prototyping critical AI systems as a situated experience.

Cited by in corpus (24)

• Expanding Explainability: Towards Social Transparency in AI systems
• "Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment
• Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens
• How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
• Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
• Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
• Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
• The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposals
• "If I Had All the Time in the World": Ophthalmologists' Perceptions of Anchoring Bias Mitigation in Clinical AI Support
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• An Uncommon Task: Participatory Design in Legal AI
• Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination Using Large Generative Models
• MedKnowts: Unified Documentation and Information Retrieval for Electronic Health Records
• Investigating Why Clinicians Deviate from Standards of Care: Liberating Patients from Mechanical Ventilation in the ICU
• Towards a Working Definition of Designing Generative User Interfaces
• "It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
• Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers
• Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
• Assembling the Puzzle: Exploring Collaboration and Data Sensemaking in Nursing Practices for Remote Patient Monitoring
• VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Models
• Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research
• Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis