SEED: Self-supervised Distillation For Visual Representation
arXiv:2101.04731
Abstract
This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, SEED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset.
Accepted as a conference paper at ICLR 2021
References in corpus (20)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- A Simple Framework for Contrastive Learning of Visual Representations
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Bootstrap your own latent: A new approach to self-supervised Learning
- Improved Baselines with Momentum Contrastive Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- FitNets: Hints for Thin Deep Nets
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Contrastive Multiview Coding
- Prototypical Contrastive Learning of Unsupervised Representations
- Big Self-Supervised Models are Strong Semi-Supervised Learners
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- Model compression via distillation and quantization
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks