Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI

Abstract

Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, sociology's interpersonal trust (i.e., trust between people). This model rests on two key properties of the vulnerability of the user and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (which detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We then present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.

arXiv commentAccepted to ACM FAccT 2021

References in corpus (14)

• Towards A Rigorous Science of Interpretable Machine Learning
• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Understanding Black-box Predictions via Influence Functions
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Towards Robust Interpretability with Self-Explaining Neural Networks
• Contrastive Explanation: A Structural-Model Approach
• Quantifying Interpretability and Trust in Machine Learning Systems
• An Interpretable Model with Globally Consistent Explanations for Credit Risk
• Mitigating Gender Bias in Natural Language Processing: Literature Review
• Towards Ecologically Valid Research on Language User Interfaces
• Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability
• ClinicalVis: Supporting Clinical Task-Focused Design Evaluation
• Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

Cited by in corpus (4)

• A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• xEEGNet: Towards Explainable AI in EEG Dementia Classification
• Saliency strikes back: How filtering out high frequencies improves white-box explanations