Improve Unsupervised Pretraining for Few-label Transfer
arXiv:2107.12369
Abstract
Unsupervised pretraining has achieved great success and many recent works have shown unsupervised pretraining can achieve comparable or even slightly better transfer performance than supervised pretraining on downstream target datasets. But in this paper, we find this conclusion may not hold when the target dataset has very few labeled samples for finetuning, \ie, few-label transfer. We analyze the possible reason from the clustering perspective: 1) The clustering quality of target samples is of great importance to few-label transfer; 2) Though contrastive learning is essential to learn how to cluster, its clustering quality is still inferior to supervised pretraining due to lack of label supervision. Based on the analysis, we interestingly discover that only involving some unlabeled target domain into the unsupervised pretraining can improve the clustering quality, subsequently reducing the transfer performance gap with supervised pretraining. This finding also motivates us to propose a new progressive few-label transfer algorithm for real applications, which aims to maximize the transfer performance under a limited annotation budget. To support our analysis and proposed method, we conduct extensive experiments on nine different target datasets. Experimental results show our proposed method can significantly boost the few-label transfer performance of unsupervised pretraining.
arXiv commentICCV 2021. arXiv admin note: substantial text overlap with arXiv:2012.05899
References in corpus (13)
- A Simple Framework for Contrastive Learning of Visual Representations
- Deep Residual Learning for Image Recognition
- Bootstrap your own latent: A new approach to self-supervised Learning
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Improved Baselines with Momentum Contrastive Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- A Simple Neural Attentive Meta-Learner
- Deep Bayesian Active Learning with Image Data
- Big Self-Supervised Models are Strong Semi-Supervised Learners
- ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
- Rethinking ImageNet Pre-training
- Intriguing Properties of Contrastive Losses
- Density-Aware Graph for Deep Semi-Supervised Visual Recognition