Adaptive Multi-Teacher Multi-level Knowledge Distillation

Abstract

Knowledge distillation~(KD) is an effective learning paradigm for improving the performance of lightweight student networks by utilizing additional supervision knowledge distilled from teacher networks. Most pioneering studies either learn from only a single teacher in their distillation learning methods, neglecting the potential that a student can learn from multiple teachers simultaneously, or simply treat each teacher to be equally important, unable to reveal the different importance of teachers for specific examples. To bridge this gap, we propose a novel adaptive multi-teacher multi-level knowledge distillation learning framework~(AMTML-KD), which consists two novel insights: (i) associating each teacher with a latent representation to adaptively learn instance-level teacher importance weights which are leveraged for acquiring integrated soft-targets~(high-level knowledge) and (ii) enabling the intermediate-level hints~(intermediate-level knowledge) to be gathered from multiple teachers by the proposed multi-group hint strategy. As such, a student model can learn multi-level knowledge from multiple teachers through AMTML-KD. Extensive results on publicly available datasets demonstrate the proposed learning framework ensures student to achieve improved performance than strong competitors.

Neurocomputing 2020.07

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Harnessing Deep Neural Networks with Logic Rules
• Multilingual Neural Machine Translation with Knowledge Distillation
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Knowledge Adaptation: Teaching to Adapt

Cited by in corpus (11)

• Knowledge Distillation: A Survey
• Bringing AI To Edge: From Deep Learning's Perspective
• A Comprehensive Survey on Imbalanced Data Learning
• Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation
• PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors
• Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data
• PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
• One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers
• Zero-shot Adversarial Quantization
• Learning When to Trust Which Teacher for Weakly Supervised ASR
• Role of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data