Variational Information Distillation for Knowledge Transfer

Abstract

Transferring knowledge from a teacher neural network pretrained on the same or a similar task to a student neural network can significantly improve the performance of the student neural network. Existing knowledge transfer approaches match the activations or the corresponding hand-crafted features of the teacher and the student networks. We propose an information-theoretic framework for knowledge transfer which formulates knowledge transfer as maximizing the mutual information between the teacher and the student networks. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10. The resulting MLP significantly outperforms the-state-of-the-art methods and it achieves similar performance to the CNN with a single convolutional layer.

To appear at CVPR 2019

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• DeepPose: Human Pose Estimation via Deep Neural Networks
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Semi-Supervised Learning with Ladder Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• How far can we go without convolution: Improving fully-connected networks

Cited by in corpus (15)

• Knowledge Distillation: A Survey
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
• Hierarchical Self-supervised Augmented Knowledge Distillation
• Learning to Disentangle Scenes for Person Re-identification
• Learning Student-Friendly Teacher Networks for Knowledge Distillation
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Distilling Knowledge via Knowledge Review
• Semi-Supervised Domain Generalizable Person Re-Identification
• Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
• Student Network Learning via Evolutionary Knowledge Distillation
• On Self-Distilling Graph Neural Network
• Learning from a Lightweight Teacher for Efficient Knowledge Distillation
• A New Training Framework for Deep Neural Network
• Midpoint Regularization: from High Uncertainty Training to Conservative Classification
• Regularization via Adaptive Pairwise Label Smoothing