Self-supervised Pre-training with Hard Examples Improves Visual Representations

Abstract

Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning. In this paper, we first present a modeling framework that unifies existing SSP methods as learning to predict pseudo-labels. Then, we propose new data augmentation methods of generating training examples whose pseudo-labels are harder to predict than those generated via random image transformations. Specifically, we use adversarial training and CutMix to create hard examples (HEXA) to be used as augmented views for MoCo-v2 and DeepCluster-v2, leading to two variants HEXA_{MoCo} and HEXA_{DCluster}, respectively. In our experiments, we pre-train models on ImageNet and evaluate them on multiple public benchmarks. Our evaluation shows that the two new algorithm variants outperform their original counterparts, and achieve new state-of-the-art on a wide range of tasks where limited task supervision is available for fine-tuning. These results verify that hard examples are instrumental in improving the generalization of the pre-trained models.

References in corpus (18)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• mixup: Beyond Empirical Risk Minimization
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Unsupervised Data Augmentation for Consistency Training
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Adversarial Training Can Hurt Generalization
• Adversarial Training for Large Neural Language Models
• Selfie: Self-supervised Pretraining for Image Embedding
• Robust Neural Machine Translation with Doubly Adversarial Inputs
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization

Cited by in corpus (7)

• Recent advances and clinical applications of deep learning in medical image analysis
• Efficient Self-supervised Vision Transformers for Representation Learning
• Few-Shot Named Entity Recognition: A Comprehensive Study
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• On Feature Decorrelation in Self-Supervised Learning
• Contrastive Attraction and Contrastive Repulsion for Representation Learning