Gradient-based Analysis of NLP Models is Manipulable
arXiv:2010.05419
Abstract
Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness. In this paper, however, we demonstrate that the gradients of a model are easily manipulable, and thus bring into question the reliability of gradient-based analyses. In particular, we merge the layers of a target model with a Facade that overwhelms the gradients without affecting the predictions. This Facade can be trained to have gradients that are misleading and irrelevant to the task, such as focusing only on the stop words in the input. On a variety of NLP tasks (text classification, NLI, and QA), we show that our method can manipulate numerous gradient-based analysis techniques: saliency maps, input reduction, and adversarial perturbations all identify unimportant or targeted tokens as being highly important. The code and a tutorial of this paper is available at http://ucinlp.github.io/facade.
References in corpus (11)
- Towards A Rigorous Science of Interpretable Machine Learning
- Axiomatic Attribution for Deep Networks
- SmoothGrad: removing noise by adding noise
- Sanity Checks for Saliency Maps
- On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
- Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
- Pathologies of Neural Models Make Interpretations Difficult
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- The (Un)reliability of saliency methods
- Fooling Neural Network Interpretations via Adversarial Model Manipulation
- How to Manipulate CNNs to Make Them Lie: the GradCAM Case