Simple Distillation Baselines for Improving Small Self-supervised Models
arXiv:2106.11304
Abstract
While large self-supervised models have rivalled the performance of their supervised counterparts, small models still struggle. In this report, we explore simple baselines for improving small self-supervised models via distillation, called SimDis. Specifically, we present an offline-distillation baseline, which establishes a new state-of-the-art, and an online-distillation baseline, which achieves similar performance with minimal computational overhead. We hope these baselines will provide useful experience for relevant future research. Code is available at: https://github.com/JindongGu/SimDis/
References in corpus (9)
- Distilling the Knowledge in a Neural Network
- A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap your own latent: A new approach to self-supervised Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- FitNets: Hints for Thin Deep Nets
- Contrastive Multiview Coding
- SEED: Self-supervised Distillation For Visual Representation
- Deep Mutual Learning
- Search for Better Students to Learn Distilled Knowledge