When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets

Abstract

Semi-Supervised Learning (SSL) has been proved to be an effective way to leverage both labeled and unlabeled data at the same time. Recent semi-supervised approaches focus on deep neural networks and have achieved promising results on several benchmarks: CIFAR10, CIFAR100 and SVHN. However, most of their experiments are based on models trained from scratch instead of pre-trained models. On the other hand, transfer learning has demonstrated its value when the target domain has limited labeled data. Here comes the intuitive question: is it possible to incorporate SSL when fine-tuning a pre-trained model? We comprehensively study how SSL methods starting from pretrained models perform under varying conditions, including training strategies, architecture choice and datasets. From this study, we obtain several interesting and useful observations. While practitioners have had an intuitive understanding of these observations, we do a comprehensive emperical analysis and demonstrate that: (1) the gains from SSL techniques over a fully-supervised baseline are smaller when trained from a pre-trained model than when trained from random initialization, (2) when the domain of the source data used to train the pre-trained model differs significantly from the domain of the target task, the gains from SSL are significantly higher and (3) some SSL methods are able to advance fully-supervised baselines (like Pseudo-Label). We hope our studies can deepen the understanding of SSL research and facilitate the process of developing more effective SSL methods to utilize pre-trained models. Code is now available at github.

Technical report

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• How transferable are features in deep neural networks?
• Wide Residual Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Ladder Networks
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• Triple Generative Adversarial Nets

Cited by in corpus (5)

• Robust Training of Social Media Image Classification Models for Rapid Disaster Response
• Adversarial Training Helps Transfer Learning via Better Representations
• Generalized Organ Segmentation by Imitating One-shot Reasoning using Anatomical Correlation
• TAGLETS: A System for Automatic Semi-Supervised Learning with Auxiliary Data
• PiaNet: A pyramid input augmented convolutional neural network for GGO detection in 3D lung CT scans