Robust Pre-Training by Adversarial Contrastive Learning
arXiv:2010.13337
Abstract
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learning representations that are consistent under both data augmentations and adversarial perturbations. Our approach leverages a recent contrastive learning framework, which learns representations by maximizing feature consistency under differently augmented views. This fits particularly well with the goal of adversarial robustness, as one cause of adversarial fragility is the lack of feature invariance, i.e., small input perturbations can result in undesirable large changes in features or even predicted labels. We explore various options to formulate the contrastive task, and demonstrate that by injecting adversarial perturbations, contrastive pre-training can lead to models that are both label-efficient and robust. We empirically evaluate the proposed Adversarial Contrastive Learning (ACL) and show it can consistently outperform existing methods. For example on the CIFAR-10 dataset, ACL outperforms the previous state-of-the-art unsupervised robust pre-training approach by 2.99% on robust accuracy and 2.14% on standard accuracy. We further demonstrate that ACL pre-training can improve semi-supervised adversarial training, even when only a few labeled examples are available. Our codes and pre-trained models have been released at: https://github.com/VITA-Group/Adversarial-Contrastive-Learning.
Neurips 2020
References in corpus (24)
- A Simple Framework for Contrastive Learning of Visual Representations
- Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
- Wide Residual Networks
- Unsupervised Data Augmentation for Consistency Training
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Momentum Contrast for Unsupervised Visual Representation Learning
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Learning Representations by Maximizing Mutual Information Across Views
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
- Contrastive Multiview Coding
- Big Self-Supervised Models are Strong Semi-Supervised Learners
- MagNet: a Two-Pronged Defense against Adversarial Examples
- Stochastic Activation Pruning for Robust Adversarial Defense
- Attacks Which Do Not Kill Training Make Adversarial Learning Stronger
- Extending Defensive Distillation
- Adversarially Robust Generalization Just Requires More Unlabeled Data
- Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
- Testing Robustness Against Unforeseen Adversaries
- Selfie: Self-supervised Pretraining for Image Embedding
- Adversarial Examples Improve Image Recognition
- Overfitting in adversarially robust deep learning
- Adversarially Robust Neural Architectures
- Intriguing properties of adversarial training at scale
- Where is the Bottleneck of Adversarial Learning with Unlabeled Data?