Do Input Gradients Highlight Discriminative Features?

Abstract

Post-hoc gradient-based interpretability methods [Simonyan et al., 2013, Smilkov et al., 2017] that provide instance-specific explanations of model predictions are often based on assumption (A): magnitude of input gradients -- gradients of logits with respect to input -- noisily highlight discriminative task-relevant features. In this work, we test the validity of assumption (A) using a three-pronged approach. First, we develop an evaluation framework, DiffROAR, to test assumption (A) on four image classification benchmarks. Our results suggest that (i) input gradients of standard models (i.e., trained on original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models satisfy (A). Second, we introduce BlockMNIST, an MNIST-based semi-real dataset, that by design encodes a priori knowledge of discriminative features. Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between input gradient attributions of standard and robust models. Finally, we theoretically prove that our empirical findings hold on a simplified version of the BlockMNIST dataset. Specifically, we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specific signal coordinates, thus grossly violating assumption (A). Our findings motivate the need to formalize and test common assumptions in interpretability in a falsifiable manner [Leavitt and Morcos, 2020]. We believe that the DiffROAR evaluation framework and BlockMNIST-based datasets can serve as sanity checks to audit instance-specific interpretability methods; code and data available at https://github.com/harshays/inputgradients.

arXiv commentNeurIPS 2021

References in corpus (26)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Interpretable machine learning: definitions, methods, and applications
• Confounding variables can degrade generalization performance of radiological deep learning models
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Sanity Checks for Saliency Maps
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Adversarial Examples Are Not Bugs, They Are Features
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Explanations can be manipulated and geometry is to blame
• A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples
• Adversarial examples from computational constraints
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• Image Synthesis with a Single (Robust) Classifier
• Benchmarking Attribution Methods with Relative Feature Importance
• Gradients of Counterfactuals
• Adversarial Robustness as a Prior for Learned Representations
• Towards falsifiable interpretability research
• Bridging Adversarial Robustness and Gradient Interpretability
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
• The shape and simplicity biases of adversarially robust ImageNet-trained CNNs
• The Dimpled Manifold Model of Adversarial Examples in Machine Learning