Representation Consolidation for Training Expert Students
arXiv:2107.08039
Abstract
Traditionally, distillation has been used to train a student model to emulate the input/output functionality of a teacher. A more useful goal than emulation, yet under-explored, is for the student to learn feature representations that transfer well to future tasks. However, we observe that standard distillation of task-specific teachers actually *reduces* the transferability of student representations to downstream tasks. We show that a multi-head, multi-task distillation method using an unlabeled proxy dataset and a generalist teacher is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance, outperforming the teacher(s) and the strong baseline of ImageNet pretrained features. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model, whose representation is improved on all teachers' domain(s).
References in corpus (14)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- Billion-scale semi-supervised learning for image classification
- Data-Free Knowledge Distillation for Deep Neural Networks
- One-Shot Federated Learning
- Data-Free Adversarial Distillation
- Zero-Shot Knowledge Distillation in Deep Networks
- FoodX-251: A Dataset for Fine-grained Food Classification
- Dream Distillation: A Data-Independent Model Compression Framework
- Large-Scale Generative Data-Free Distillation
- Scalable Transfer Learning with Expert Models
- Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN
- Knowledge Flow: Improve Upon Your Teachers
- A linearized framework and a new benchmark for model selection for fine-tuning