How model accuracy and explanation fidelity influence user trust

Abstract

Machine learning systems have become popular in fields such as marketing, financing, or data mining. While they are highly accurate, complex machine learning systems pose challenges for engineers and users. Their inherent complexity makes it impossible to easily judge their fairness and the correctness of statistically learned relations between variables and classes. Explainable AI aims to solve this challenge by modelling explanations alongside with the classifiers, potentially improving user trust and acceptance. However, users should not be fooled by persuasive, yet untruthful explanations. We therefore conduct a user study in which we investigate the effects of model accuracy and explanation fidelity, i.e. how truthfully the explanation represents the underlying model, on user trust. Our findings show that accuracy is more important for user trust than explainability. Adding an explanation for a classification result can potentially harm trust, e.g. when adding nonsensical explanations. We also found that users cannot be tricked by high-fidelity explanations into having trust for a bad classifier. Furthermore, we found a mismatch between observed (implicit) and self-reported (explicit) trust.

AI IJCAI Workshop on Explainable Artificial Intelligence (X-AI) 2019

References in corpus (2)

• Pathologies of Neural Models Make Interpretations Difficult
• Generating Counterfactual Explanations with Natural Language

Cited by in corpus (10)

• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations
• Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
• A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning
• AI for Explaining Decisions in Multi-Agent Environments
• Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutions
• Do Not Trust a Model Because It is Confident: Uncovering and Characterizing Unknown Unknowns to Student Success Predictors in Online-Based Learning
• Evaluating the Explainability of Attributes and Prototypes for a Medical Classification Model
• Data in context: How digital transformation can support human reasoning in cyber-physical production systems
• Extracting Optimal Explanations for Ensemble Trees via Logical Reasoning