Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations

Abstract

Recent progress in self-supervised learning has demonstrated promising results in multiple visual tasks. An important ingredient in high-performing self-supervised methods is the use of data augmentation by training models to place different augmented views of the same image nearby in embedding space. However, commonly used augmentation pipelines treat images holistically, ignoring the semantic relevance of parts of an image-e.g. a subject vs. a background-which can lead to the learning of spurious correlations. Our work addresses this problem by investigating a class of simple, yet highly effective "background augmentations", which encourage models to focus on semantically-relevant content by discouraging them from focusing on image backgrounds. Through a systematic investigation, we show that background augmentations lead to substantial improvements in performance across a spectrum of state-of-the-art self-supervised methods (MoCo-v2, BYOL, SwAV) on a variety of tasks, e.g. $\sim$+1-2% gains on ImageNet, enabling performance on par with the supervised baseline. Further, we find the improvement in limited-labels settings is even larger (up to 4.2%). Background augmentations also improve robustness to a number of distribution shifts, including natural adversarial examples, ImageNet-9, adversarial attacks, ImageNet-Renditions. We also make progress in completely unsupervised saliency detection, in the process of generating saliency masks used for background augmentations.

arXiv commentTechnical Report; Additional Results

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• U${}^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• What Makes for Good Views for Contrastive Learning?
• DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection
• Large Batch Training of Convolutional Networks
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Exploring Simple Siamese Representation Learning
• Do ImageNet Classifiers Generalize to ImageNet?
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Natural Adversarial Examples
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Efficient Self-supervised Vision Transformers for Representation Learning
• Are all negatives created equal in contrastive instance discrimination?
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• Time for a Background Check! Uncovering the impact of Background Features on Deep Neural Networks
• DeepUSPS: Deep Robust Unsupervised Saliency Prediction With Self-Supervision