Sanity Simulations for Saliency Methods

Abstract

Saliency methods are a popular class of feature attribution explanation methods that aim to capture a model's predictive reasoning by identifying "important" pixels in an input image. However, the development and adoption of these methods are hindered by the lack of access to ground-truth model reasoning, which prevents accurate evaluation. In this work, we design a synthetic benchmarking framework, SMERF, that allows us to perform ground-truth-based evaluation while controlling the complexity of the model's reasoning. Experimentally, SMERF reveals significant limitations in existing saliency methods and, as a result, represents a useful tool for the development of new saliency methods.

arXiv commentAccepted to International Conference on Machine Learning (ICML 2022)

References in corpus (10)

• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• Towards Robust Interpretability with Self-Explaining Neural Networks
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Benchmarking Attribution Methods with Relative Feature Importance
• Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging