Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning

Abstract

Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We find these robust pre-trained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3.83% on robust accuracy and 1.3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline. Moreover, we find that different self-supervised pre-trained models have a diverse adversarial vulnerability. It inspires us to ensemble several pretraining tasks, which boosts robustness more. Our ensemble strategy contributes to a further improvement of 3.59% on robust accuracy, while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.

arXiv commentCVPR 2020

References in corpus (18)

• Ensemble Adversarial Training: Attacks and Defenses
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• Adversarially Robust Generalization Requires More Data
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Stochastic Activation Pruning for Robust Adversarial Defense
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Are Labels Required for Improving Adversarial Robustness?
• Image Synthesis with a Single (Robust) Classifier
• Extending Defensive Distillation
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Testing Robustness Against Unforeseen Adversaries
• Selfie: Self-supervised Pretraining for Image Embedding
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• Unlabeled Data Improves Adversarial Robustness
• Model Compression with Adversarial Robustness: A Unified Optimization Framework
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• Towards Understanding the Transferability of Deep Representations

Cited by in corpus (11)

• Graph Contrastive Learning with Augmentations
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• A Closer Look at Accuracy vs. Robustness
• When Does Self-Supervision Help Graph Convolutional Networks?
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Geometry-aware Instance-reweighted Adversarial Training
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
• Reliable Adversarial Distillation with Unreliable Teachers
• Probabilistic Margins for Instance Reweighting in Adversarial Training
• What You See is Not What the Network Infers: Detecting Adversarial Examples Based on Semantic Contradiction
• 10 Security and Privacy Problems in Large Foundation Models