Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making

Abstract

Although AI holds promise for improving human decision making in societally critical domains, it remains an open question how human-AI teams can reliably outperform AI alone and human alone in challenging prediction tasks (also known as complementary performance). We explore two directions to understand the gaps in achieving complementary performance. First, we argue that the typical experimental setup limits the potential of human-AI teams. To account for lower AI performance out-of-distribution than in-distribution because of distribution shift, we design experiments with different distribution types and investigate human performance for both in-distribution and out-of-distribution examples. Second, we develop novel interfaces to support interactive explanations so that humans can actively engage with AI assistance. Using virtual pilot studies and large-scale randomized experiments across three tasks, we demonstrate a clear difference between in-distribution and out-of-distribution, and observe mixed results for interactive explanations: while interactive explanations improve human perception of AI assistance's usefulness, they may reinforce human biases and lead to limited performance improvement. Overall, our work points out critical challenges and future directions towards enhancing human performance with AI assistance.

arXiv comment45 pages, 24 figures, accepted to CSCW 2021

References in corpus (4)

• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
• Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
• Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes

Cited by in corpus (10)

• When combinations of humans and AI are useful: A systematic review and meta-analysis
• "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
• Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
• Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
• Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
• Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
• Beyond Recommendations: From Backward to Forward AI Support of Pilots' Decision-Making Process
• The Impact of Explanations on Fairness in Human-AI Decision-Making: Protected vs Proxy Features
• Give Me a Choice: The Consequences of Restricting Choices Through AI-Support for Perceived Autonomy, Motivational Variables, and Decision Performance
• Design Considerations for Human Oversight of AI: Insights from Co-Design Workshops and Work Design Theory