"Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations

Abstract

Methods for interpreting machine learning black-box models increase the outcomes' transparency and in turn generates insight into the reliability and fairness of the algorithms. However, the interpretations themselves could contain significant uncertainty that undermines the trust in the outcomes and raises concern about the model's reliability. Focusing on the method "Local Interpretable Model-agnostic Explanations" (LIME), we demonstrate the presence of two sources of uncertainty, namely the randomness in its sampling procedure and the variation of interpretation quality across different input data points. Such uncertainty is present even in models with high training and test accuracy. We apply LIME to synthetic data and two public data sets, text classification in 20 Newsgroup and recidivism risk-scoring in COMPAS, to support our argument.

References in corpus (9)

• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Predictive learning via rule ensembles
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• How to Explain Individual Classification Decisions
• Sanity Checks for Saliency Maps
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier

Cited by in corpus (14)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• On Interpretability of Artificial Neural Networks: A Survey
• Generating collective counterfactual explanations in score-based classification via mathematical optimization
• Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
• A study of data and label shift in the LIME framework
• bLIMEy: Surrogate Prediction Explanations Beyond LIME
• IROF: a low resource evaluation metric for explanation methods
• What will it take to generate fairness-preserving explanations?
• Generative Perturbation Analysis for Probabilistic Black-Box Anomaly Attribution
• Show or Suppress? Managing Input Uncertainty in Machine Learning Model Explanations
• Information-theoretic Evolution of Model Agnostic Global Explanations