Do Adversarially Robust ImageNet Models Transfer Better?

Abstract

Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations. Our code and models are available at https://github.com/Microsoft/robust-models-transfer .

arXiv commentNeurIPS 2020

References in corpus (26)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• Understanding Black-box Predictions via Influence Functions
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Certified Defenses against Adversarial Examples
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• What makes ImageNet good for transfer learning?
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Adversarial Training Can Hurt Generalization
• Adversarial examples from computational constraints
• Image Synthesis with a Single (Robust) Classifier
• Interpreting Adversarially Trained Convolutional Neural Networks
• Adversarial Robustness as a Prior for Learned Representations
• Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets
• Adversarial camera stickers: A physical camera-based attack on deep learning systems
• Convergence of Adversarial Training in Overparametrized Neural Networks
• Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?
• Randomized Smoothing of All Shapes and Sizes
• Bridging Adversarial Robustness and Gradient Interpretability
• On the Benefits of Models with Perceptually-Aligned Gradients

Cited by in corpus (10)

• Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow
• ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches
• Exposing Previously Undetectable Faults in Deep Neural Networks
• Towards Resilient Artificial Intelligence: Survey and Research Issues
• An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Networks
• Improved OOD Generalization via Adversarial Training and Pre-training
• Adversarially Trained Object Detector for Unsupervised Domain Adaptation
• Language Guided Adversarial Purification
• Energy Attack: On Transferring Adversarial Examples
• Adversarial Defenses via Vector Quantization