Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes

Abstract

Clinical decision support tools (DST) promise improved healthcare outcomes by offering data-driven insights. While effective in lab settings, almost all DSTs have failed in practice. Empirical research diagnosed poor contextual fit as the cause. This paper describes the design and field evaluation of a radically new form of DST. It automatically generates slides for clinicians' decision meetings with subtly embedded machine prognostics. This design took inspiration from the notion of "Unremarkable Computing", that by augmenting the users' routines technology/AI can have significant importance for the users yet remain unobtrusive. Our field evaluation suggests clinicians are more likely to encounter and embrace such a DST. Drawing on their responses, we discuss the importance and intricacies of finding the right level of unremarkableness in DST design, and share lessons learned in prototyping critical AI systems as a situated experience.

Cited by in corpus (34)

• Expanding Explainability: Towards Social Transparency in AI systems
• "Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment
• Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens
• How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
• Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making
• Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis
• Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
• Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
• Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
• Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists
• Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models
• The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposals
• "If I Had All the Time in the World": Ophthalmologists' Perceptions of Anchoring Bias Mitigation in Clinical AI Support
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• An Uncommon Task: Participatory Design in Legal AI
• Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination Using Large Generative Models
• MedKnowts: Unified Documentation and Information Retrieval for Electronic Health Records
• Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI
• Investigating Why Clinicians Deviate from Standards of Care: Liberating Patients from Mechanical Ventilation in the ICU
• REX: Designing User-centered Repair and Explanations to Address Robot Failures
• Towards a Working Definition of Designing Generative User Interfaces
• Summaries, Highlights, and Action items: Design, implementation and evaluation of an LLM-powered meeting recap system
• "It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
• Beyond Recommendations: From Backward to Forward AI Support of Pilots' Decision-Making Process
• Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
• Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers
• Assembling the Puzzle: Exploring Collaboration and Data Sensemaking in Nursing Practices for Remote Patient Monitoring
• VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Models
• Sensemaking Through Making: Developing Clinical Domain Knowledge by Crafting Synthetic Datasets and Prototyping System Architectures
• Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis
• Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research
• Crystallizing Schemas with Teleoscope: Thematic Curation of Large Text Corpora on Reddit