Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

Abstract

While recent studies on semi-supervised learning have shown remarkable progress in leveraging both labeled and unlabeled data, most of them presume a basic setting of the model is randomly initialized. In this work, we consider semi-supervised learning and transfer learning jointly, leading to a more practical and competitive paradigm that can utilize both powerful pre-trained models from source domain as well as labeled/unlabeled data in the target domain. To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization that consists of two complementary components: Adaptive Knowledge Consistency (AKC) on the examples between the source and target model, and Adaptive Representation Consistency (ARC) on the target model between labeled and unlabeled examples. Examples involved in the consistency regularization are adaptively selected according to their potential contributions to the target task. We conduct extensive experiments on popular benchmarks including CIFAR-10, CUB-200, and MURA, by fine-tuning the ImageNet pre-trained ResNet-50 model. Results show that our proposed adaptive consistency regularization outperforms state-of-the-art semi-supervised learning techniques such as Pseudo Label, Mean Teacher, and FixMatch. Moreover, our algorithm is orthogonal to existing methods and thus able to gain additional improvements on top of MixMatch and FixMatch. Our code is available at https://github.com/SHI-Labs/Semi-Supervised-Transfer-Learning.

arXiv commentCVPR 2021

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Prototypical Networks for Few-shot Learning
• How transferable are features in deep neural networks?
• Learning Transferable Features with Deep Adaptation Networks
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• Interpolation Consistency Training for Semi-Supervised Learning
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• What makes ImageNet good for transfer learning?
• DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks
• Domain Adaptive Transfer Learning with Specialist Models
• RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr