Selfie: Self-supervised Pretraining for Image Embedding

Abstract

We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-out patches in an input image, our method learns to select the correct patch, among other "distractor" patches sampled from the same image, to fill in the masked location. This classification objective sidesteps the need for predicting exact pixel values of the target patches. The pretraining architecture of Selfie includes a network of convolutional blocks to process patches followed by an attention pooling network to summarize the content of unmasked patches before predicting masked ones. During finetuning, we reuse the convolutional weights found by pretraining. We evaluate Selfie on three benchmarks (CIFAR-10, ImageNet 32 x 32, and ImageNet 224 x 224) with varying amounts of labeled data, from 5% to 100% of the training sets. Our pretraining method provides consistent improvements to ResNet-50 across all settings compared to the standard supervised training of the same network. Notably, on ImageNet 224 x 224 with 60 examples per class (5%), our method improves the mean accuracy of ResNet-50 from 35.6% to 46.7%, an improvement of 11.1 points in absolute accuracy. Our pretraining method also improves ResNet-50 training stability, especially on low data regime, by significantly lowering the standard deviation of test accuracies across different runs.

References in corpus (9)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Distributed Representations of Sentences and Documents
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Data Augmentation for Consistency Training
• Temporal Ensembling for Semi-Supervised Learning
• Interpolation Consistency Training for Semi-Supervised Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Skip-Thought Vectors
• Revisiting Self-Supervised Visual Representation Learning

Cited by in corpus (27)

• Unsupervised Data Augmentation for Consistency Training
• BEiT: BERT Pre-Training of Image Transformers
• Graph Contrastive Learning with Augmentations
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• UNITER: UNiversal Image-TExt Representation Learning
• Efficient Self-supervised Vision Transformers for Representation Learning
• Robust Pre-Training by Adversarial Contrastive Learning
• When Does Self-Supervision Help Graph Convolutional Networks?
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Contrastive Multi-Level Graph Neural Networks for Session-based Recommendation
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
• Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• Self-supervised Pre-training with Hard Examples Improves Visual Representations
• SimMIM: A Simple Framework for Masked Image Modeling
• When Does Self-supervision Improve Few-shot Learning?
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Towards Domain-Agnostic Contrastive Learning
• Rethinking Supervised Pre-training for Better Downstream Transferring
• Towards a Hypothesis on Visual Transformation based Self-Supervision
• Geographical Knowledge-driven Representation Learning for Remote Sensing Images
• Investigate the Essence of Long-Tailed Recognition from a Unified Perspective
• Instance Separation Emerges from Inpainting
• OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning
• InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping