ImageNet Pre-training also Transfers Non-Robustness

Abstract

ImageNet pre-training has enabled state-of-the-art results on many tasks. In spite of its recognized contribution to generalization, we observed in this study that ImageNet pre-training also transfers adversarial non-robustness from pre-trained model into fine-tuned model in the downstream classification tasks. We first conducted experiments on various datasets and network backbones to uncover the adversarial non-robustness in fine-tuned model. Further analysis was conducted on examining the learned knowledge of fine-tuned model and standard model, and revealed that the reason leading to the non-robustness is the non-robust features transferred from ImageNet pre-trained model. Finally, we analyzed the preference for feature learning of the pre-trained model, explored the factors influencing robustness, and introduced a simple robust ImageNet pre-training solution. Our code is available at \url{https://github.com/jiamingzhang94/ImageNet-Pretraining-transfers-non-robustness}.

Accepted by AAAI2023

References in corpus (9)

• How transferable are features in deep neural networks?
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Shortcut Learning in Deep Neural Networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• Similarity of Neural Network Representations Revisited
• Med3D: Transfer Learning for 3D Medical Image Analysis
• Billion-scale semi-supervised learning for image classification
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability