Comparing Zealous and Restrained AI Recommendations in a Real-World Human-AI Collaboration Task

Abstract

When designing an AI-assisted decision-making system, there is often a tradeoff between precision and recall in the AI's recommendations. We argue that careful exploitation of this tradeoff can harness the complementary strengths in the human-AI collaboration to significantly improve team performance. We investigate a real-world video anonymization task for which recall is paramount and more costly to improve. We analyze the performance of 78 professional annotators working with a) no AI assistance, b) a high-precision "restrained" AI, and c) a high-recall "zealous" AI in over 3,466 person-hours of annotation work. In comparison, the zealous AI helps human teammates achieve significantly shorter task completion time and higher recall. In a follow-up study, we remove AI assistance for everyone and find negative training effects on annotators trained with the restrained AI. These findings and our analysis point to important implications for the design of AI assistance in recall-demanding scenarios.

arXiv comment15 pages, 14 figures, accepted to ACM CHI 2023

References in corpus (4)

• The VIA Annotation Software for Images, Audio and Video
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysis
• Ego4D: Around the World in 3,000 Hours of Egocentric Video

Cited by in corpus (2)

• Interaction Configurations and Prompt Guidance in Conversational AI for Question Answering in Human-AI Teams
• A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health