Benchmarking Attribution Methods with Relative Feature Importance

Abstract

Interpretability is an important area of research for safe deployment of machine learning systems. One particular type of interpretability method attributes model decisions to input features. Despite active development, quantitative evaluation of feature attribution methods remains difficult due to the lack of ground truth: we do not know which input features are in fact important to a model. In this work, we propose a framework for Benchmarking Attribution Methods (BAM) with a priori knowledge of relative feature importance. BAM includes 1) a carefully crafted dataset and models trained with known relative feature importance and 2) three complementary metrics to quantitatively evaluate attribution methods by comparing feature attributions between pairs of models and pairs of inputs. Our evaluation on several widely-used attribution methods suggests that certain methods are more likely to produce false positive explanations---features that are incorrectly attributed as more important to model prediction. We open source our dataset, models, and metrics.

References in corpus (13)

• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Striving for Simplicity: The All Convolutional Net
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Sanity Checks for Saliency Maps
• Towards Robust Interpretability with Self-Explaining Neural Networks
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
• Interpretation of Neural Networks is Fragile
• Adversarial Robustness as a Prior for Learned Representations
• Excessive Invariance Causes Adversarial Vulnerability
• ImageNet Large Scale Visual Recognition Challenge

Cited by in corpus (24)

• Captum: A unified and generic model interpretability library for PyTorch
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Explaining Classifiers with Causal Concept Effect (CaCE)
• Explaining Anomalies Detected by Autoencoders Using SHAP
• Where is the Model Looking At?--Concentrate and Explain the Network Attention
• Synthetic Benchmarks for Scientific Research in Explainable Machine Learning
• Quantitative Evaluations on Saliency Methods: An Experimental Study
• Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification
• Towards a Unified Evaluation of Explanation Methods without Ground Truth
• A Survey on the Robustness of Feature Importance and Counterfactual Explanations
• Towards Ground Truth Explainability on Tabular Data
• Improving Interpretability in Medical Imaging Diagnosis using Adversarial Training
• On The Coherence of Quantitative Evaluation of Visual Explanations
• Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
• A Novel Hybrid Feature Importance and Feature Interaction Detection Framework for Predictive Optimization in Industry 4.0 Applications
• Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing
• Measuring and improving the quality of visual explanations
• Robust Semantic Interpretability: Revisiting Concept Activation Vectors
• Do Input Gradients Highlight Discriminative Features?
• Debiasing Convolutional Neural Networks via Meta Orthogonalization
• Evaluating Attribution Methods using White-Box LSTMs
• Longitudinal Distance: Towards Accountable Instance Attribution
• Sanity Simulations for Saliency Methods
• Explainability Requires Interactivity