An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness

Abstract

Deep neural networks (DNNs) have had many successes, but they suffer from two major issues: (1) a vulnerability to adversarial examples and (2) a tendency to elude human interpretation. Interestingly, recent empirical and theoretical evidence suggests these two seemingly disparate issues are actually connected. In particular, robust models tend to provide more interpretable gradients than non-robust models. However, whether this relationship works in the opposite direction remains obscure. With this paper, we seek empirical answers to the following question: can models acquire adversarial robustness when they are trained to have interpretable gradients? We introduce a theoretically inspired technique called Interpretation Regularization (IR), which encourages a model's gradients to (1) match the direction of interpretable target salience maps and (2) have small magnitude. To assess model performance and tease apart factors that contribute to adversarial robustness, we conduct extensive experiments on MNIST and CIFAR-10 with both ${\ell }_2$ and ${\ell }_{\infty }$ attacks. We demonstrate that training the networks to have interpretable gradients improves their robustness to adversarial perturbations. Applying the network interpretation technique SmoothGrad yields additional performance gains, especially in cross-norm attacks and under heavy perturbations. The results indicate that the interpretability of the model gradients is a crucial factor for adversarial robustness. Code for the experiments can be found at https://github.com/a1noack/interp_regularization.

arXiv commentAccepted by the journal Springer Nature Computer Science

References in corpus (29)

• Axiomatic Attribution for Deep Networks
• Methods for Interpreting and Understanding Deep Neural Networks
• Learning Important Features Through Propagating Activation Differences
• Ensemble Adversarial Training: Attacks and Defenses
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Understanding Black-box Predictions via Influence Functions
• Delving into Transferable Adversarial Examples and Black-box Attacks
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Sanity Checks for Saliency Maps
• On Evaluating Adversarial Robustness
• Adversarially Robust Generalization Requires More Data
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• On the (Statistical) Detection of Adversarial Examples
• Black-box Adversarial Attacks with Limited Queries and Information
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Robust Large Margin Deep Neural Networks
• A mathematical theory of semantic development in deep neural networks
• Adversarial examples from computational constraints
• Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples
• Adversarial Robustness May Be at Odds With Simplicity
• First-order Adversarial Vulnerability of Neural Networks and Input Dimension
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Robust Learning with Jacobian Regularization
• NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks
• On the Connection Between Adversarial Robustness and Saliency Map Interpretability
• Quantifying the Preferential Direction of the Model Gradient in Adversarial Training With Projected Gradient Descent