Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation

Abstract

Knowledge Distillation is becoming one of the primary trends among neural network compression algorithms to improve the generalization performance of a smaller student model with guidance from a larger teacher model. This momentous rise in applications of knowledge distillation is accompanied by the introduction of numerous algorithms for distilling the knowledge such as soft targets and hint layers. Despite this advancement in different techniques for distilling the knowledge, the aggregation of different paths for distillation has not been studied comprehensively. This is of particular significance, not only because different paths have different importance, but also due to the fact that some paths might have negative effects on the generalization performance of the student model. Hence, we need to adaptively adjust the importance of each path to maximize the impact of distillation on the student model. In this paper, we explore different approaches for aggregating these different paths and introduce our proposed adaptive approach based on multitask learning methods. We empirically demonstrate the effectiveness of the proposed approach over other baselines on the applications of knowledge distillation in classification, semantic segmentation, and object detection tasks.

Accepted to BMVC 2021 for publication. V2. Added more results for ImageNet-1K

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Knowledge Distillation: A Survey
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Multi-Task Learning as Multi-Objective Optimization
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• What is the State of Neural Network Pruning?
• Pareto Multi-Task Learning
• AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving
• Knowledge Flow: Improve Upon Your Teachers
• Localization Distillation for Object Detection
• Does Knowledge Distillation Really Work?
• Pareto Efficient Fairness in Supervised Learning: From Extraction to Tracing