Saliency is a Possible Red Herring When Diagnosing Poor Generalization
arXiv:1910.00199
Abstract
Poor generalization is one symptom of models that learn to predict target variables using spuriously-correlated image features present only in the training distribution instead of the true image features that denote a class. It is often thought that this can be diagnosed visually using attribution (aka saliency) maps. We study if this assumption is correct. In some prediction tasks, such as for medical images, one may have some images with masks drawn by a human expert, indicating a region of the image containing relevant information to make the prediction. We study multiple methods that take advantage of such auxiliary labels, by training networks to ignore distracting features which may be found outside of the region of interest. This mask information is only used during training and has an impact on generalization accuracy depending on the severity of the shift between the training and test distributions. Surprisingly, while these methods improve generalization performance in the presence of a covariate shift, there is no strong correspondence between the correction of attribution towards the features a human expert has labelled as important and generalization performance. These results suggest that the root cause of poor generalization may not always be spatially defined, and raise questions about the utility of masks as "attribution priors" as well as saliency maps for explainable predictions.
25 pages, 27 figures, 5 tables, code in paper (https://github.com/josephdviviano/saliency-red-herring). Published at International Conference on Learning Representations (ICLR) 2021. Previously titled "Underwhelming Generalization Improvements from Controlling Feature Attribution"
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- Axiomatic Attribution for Deep Networks
- Improved Regularization of Convolutional Neural Networks with Cutout
- Shortcut Learning in Deep Neural Networks
- Confounding variables can degrade generalization performance of radiological deep learning models
- Understanding deep learning requires rethinking generalization
- A large annotated medical image dataset for the development and evaluation of segmentation algorithms
- PadChest: A large chest x-ray image dataset with multi-label annotated reports
- Captum: A unified and generic model interpretability library for PyTorch
- What do we need to build explainable AI systems for the medical domain?
- A review of domain adaptation without target labels
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Invariant Risk Minimization
- Measuring the tendency of CNNs to Learn Surface Statistical Regularities
- Deep neural network or dermatologist?
- Learning explanations that are hard to vary
- What shapes feature representations? Exploring datasets, architectures, and training
- Subspace Alignment For Domain Adaptation
- Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
- GradMask: Reduce Overfitting by Regularizing Saliency
- Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps
- Confounder-Aware Visualization of ConvNets