What it Thinks is Important is Important: Robustness Transfers through Input Gradients

Abstract

Adversarial perturbations are imperceptible changes to input pixels that can change the prediction of deep learning models. Learned weights of models robust to such perturbations are previously found to be transferable across different tasks but this applies only if the model architecture for the source and target tasks is the same. Input gradients characterize how small changes at each input pixel affect the model output. Using only natural images, we show here that training a student model's input gradients to match those of a robust teacher model can gain robustness close to a strong baseline that is robustly trained from scratch. Through experiments in MNIST, CIFAR-10, CIFAR-100 and Tiny-ImageNet, we show that our proposed method, input gradient adversarial matching, can transfer robustness across different tasks and even across different model architectures. This demonstrates that directly targeting the semantics of input gradients is a feasible way towards adversarial robustness.

arXiv commentAccepted as Oral in CVPR 2020, Camera-Ready Version

References in corpus (17)

• Towards Deep Learning Models Resistant to Adversarial Attacks
• Theoretically Principled Trade-off between Robustness and Accuracy
• Autoencoding beyond pixels using a learned similarity metric
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Robustness May Be at Odds with Accuracy
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Adversarial Robustness through Local Linearization
• Semidefinite relaxations for certifying robustness to adversarial examples
• Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
• Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack
• Provably Robust Boosted Decision Stumps and Trees against Adversarial Attacks
• Fixing the train-test resolution discrepancy
• COCO-GAN: Generation by Parts via Conditional Coordinating
• Jacobian Adversarially Regularized Networks for Robustness