How Useful is Self-Supervised Pretraining for Visual Tasks?
arXiv:2003.14323
Abstract
Recent advances have spurred incredible progress in self-supervised pretraining for vision. We investigate what factors may play a role in the utility of these pretraining methods for practitioners. To do this, we evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks. We prepare a suite of synthetic data that enables an endless supply of annotated images as well as full control over dataset difficulty. Our experiments offer insights into how the utility of self-supervision changes as the number of available labels grows as well as how the utility changes as a function of the downstream task and the properties of the training data. We also find that linear evaluation does not correlate with finetuning performance. Code and data is available at \href{https://www.github.com/princeton-vl/selfstudy}{github.com/princeton-vl/selfstudy}.
arXiv commentTo appear in CVPR 2020
References in corpus (18)
- Representation Learning with Contrastive Predictive Coding
- Improved Regularization of Convolutional Neural Networks with Cutout
- Momentum Contrast for Unsupervised Visual Representation Learning
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- Learning Representations by Maximizing Mutual Information Across Views
- Tune: A Research Platform for Distributed Model Selection and Training
- MixMatch: A Holistic Approach to Semi-Supervised Learning
- Contrastive Multiview Coding
- Billion-scale semi-supervised learning for image classification
- Self-training with Noisy Student improves ImageNet classification
- Unsupervised Learning by Predicting Noise
- Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
- Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
- Local Aggregation for Unsupervised Learning of Visual Embeddings
- Revisiting Self-Supervised Visual Representation Learning
- Probing the State of the Art: A Critical Look at Visual Representation Evaluation