Self-supervised Knowledge Distillation for Few-shot Learning

Abstract

Real-world contains an overwhelmingly large number of object classes, learning all of which at once is infeasible. Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. Recent works [7, 41] show that simply learning a good feature embedding can outperform more sophisticated meta-learning and metric learning algorithms for few-shot learning. In this paper, we propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. We follow a two-stage learning process: First, we train a neural network to maximize the entropy of the feature embedding, thus creating an optimal output manifold using a self-supervised auxiliary loss. In the second stage, we minimize the entropy on feature embedding by bringing self-supervised twins together, while constraining the manifold with student-teacher distillation. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods, with further gains achieved by our second stage distillation process. Our codes are available at: https://github.com/brjathu/SKD.

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Matching Networks for One Shot Learning
• A Simple Neural Attentive Meta-Learner
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Unsupervised Domain Adaptation through Self-Supervision
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Infinite Mixture Prototypes for Few-Shot Learning
• Transductive Episodic-Wise Adaptive Metric for Few-Shot Learning
• Few-Shot Learning with Global Class Representations

Cited by in corpus (9)

• The Art of Camouflage: Few-Shot Learning for Animal Detection and Segmentation
• LibFewShot: A Comprehensive Library for Few-shot Learning
• Temperature as Uncertainty in Contrastive Learning
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback
• S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• Enabling the Network to Surf the Internet
• Few-shot Learning with Global Relatedness Decoupled-Distillation
• Towards Data-Free Domain Generalization