A Study of the Generalizability of Self-Supervised Representations
arXiv:2109.09150 · doi:10.1016/j.mlwa.2021.100124
Abstract
Recent advancements in self-supervised learning (SSL) made it possible to learn generalizable visual representations from unlabeled data. The performance of Deep Learning models fine-tuned on pretrained SSL representations is on par with models fine-tuned on the state-of-the-art supervised learning (SL) representations. Irrespective of the progress made in SSL, its generalizability has not been studied extensively. In this article, we perform a deeper analysis of the generalizability of pretrained SSL and SL representations by conducting a domain-based study for transfer learning classification tasks. The representations are learned from the ImageNet source data, which are then fine-tuned using two types of target datasets: similar to the source dataset, and significantly different from the source dataset. We study generalizability of the SSL and SL-based models via their prediction accuracy as well as prediction confidence. In addition to this, we analyze the attribution of the final convolutional layer of these models to understand how they reason about the semantic identity of the data. We show that the SSL representations are more generalizable as compared to the SL representations. We explain the generalizability of the SSL representations by investigating its invariance property, which is shown to be better than that observed in the SL representations.
Journal of Machine Learning With Applications (MLWA)
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Generative Adversarial Networks
- How transferable are features in deep neural networks?
- Bootstrap your own latent: A new approach to self-supervised Learning
- Understanding Neural Networks Through Deep Visualization
- What makes ImageNet good for transfer learning?
- Self-supervised Pretraining of Visual Features in the Wild
- The Devil is in the Tails: Fine-grained Classification in the Wild
- Learning Invariant Representations with Local Transformations