A Study of the Generalizability of Self-Supervised Representations

Abstract

Recent advancements in self-supervised learning (SSL) made it possible to learn generalizable visual representations from unlabeled data. The performance of Deep Learning models fine-tuned on pretrained SSL representations is on par with models fine-tuned on the state-of-the-art supervised learning (SL) representations. Irrespective of the progress made in SSL, its generalizability has not been studied extensively. In this article, we perform a deeper analysis of the generalizability of pretrained SSL and SL representations by conducting a domain-based study for transfer learning classification tasks. The representations are learned from the ImageNet source data, which are then fine-tuned using two types of target datasets: similar to the source dataset, and significantly different from the source dataset. We study generalizability of the SSL and SL-based models via their prediction accuracy as well as prediction confidence. In addition to this, we analyze the attribution of the final convolutional layer of these models to understand how they reason about the semantic identity of the data. We show that the SSL representations are more generalizable as compared to the SL representations. We explain the generalizability of the SSL representations by investigating its invariance property, which is shown to be better than that observed in the SL representations.

arXiv commentJournal of Machine Learning With Applications (MLWA)

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Generative Adversarial Networks
• How transferable are features in deep neural networks?
• Bootstrap your own latent: A new approach to self-supervised Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Unsupervised Representation Learning by Predicting Image Rotations
• Understanding Neural Networks Through Deep Visualization
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Rethinking Pre-training and Self-training
• Deep Generative Stochastic Networks Trainable by Backprop
• What makes ImageNet good for transfer learning?
• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
• Self-supervised Pretraining of Visual Features in the Wild
• The Devil is in the Tails: Fine-grained Classification in the Wild
• Learning Invariant Representations with Local Transformations
• Learning Representations for Automatic Colorization
• What makes instance discrimination good for transfer learning?

Cited by in corpus (6)

• Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Self-supervision for medical image classification: state-of-the-art performance with ~100 labeled training samples per class
• Self-Supervised Learning for Text Recognition: A Critical Survey
• Structure-preserving contrastive learning for spatial time series
• Navigating the Kaleidoscope of COVID-19 Misinformation Using Deep Learning