Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing

Abstract

While enjoying the great achievements brought by deep learning (DL), people are also worried about the decision made by DL models, since the high degree of non-linearity of DL models makes the decision extremely difficult to understand. Consequently, attacks such as adversarial attacks are easy to carry out, but difficult to detect and explain, which has led to a boom in the research on local explanation methods for explaining model decisions. In this paper, we evaluate the faithfulness of explanation methods and find that traditional tests on faithfulness encounter the random dominance problem, \ie, the random selection performs the best, especially for complex data. To further solve this problem, we propose three trend-based faithfulness tests and empirically demonstrate that the new trend tests can better assess faithfulness than traditional tests on image, natural language and security tasks. We implement the assessment system and evaluate ten popular explanation methods. Benefiting from the trend tests, we successfully assess the explanation methods on complex data for the first time, bringing unprecedented discoveries and inspiring future research. Downstream tasks also greatly benefit from the tests. For example, model debugging equipped with faithful explanation methods performs much better for detecting and correcting accuracy and security problems.

References in corpus (20)

• Sequence to Sequence Learning with Neural Networks
• Explaining and Harnessing Adversarial Examples
• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Captum: A unified and generic model interpretability library for PyTorch
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements
• Explanations can be manipulated and geometry is to blame
• Benchmarking Attribution Methods with Relative Feature Importance
• Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers
• Understanding the Failure Modes of Out-of-Distribution Generalization
• Adversarial Robustness of Deep Code Comment Generation
• Human-in-the-Loop Interpretability Prior
• Deep Learning Through the Lens of Example Difficulty
• Hidden Backdoor Attack against Semantic Segmentation Models
• Understanding the unstable convergence of gradient descent