FitNets: Hints for Thin Deep Nets

Abstract

While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Going Deeper with Convolutions
• Theano: new features and speed improvements
• Compressing Deep Convolutional Networks using Vector Quantization
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Cited by in corpus (29)

• Residual Networks of Residual Networks: Multilevel Residual Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Data-Free Knowledge Distillation for Deep Neural Networks
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• The Importance of Skip Connections in Biomedical Image Segmentation
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Focus: Querying Large Video Datasets with Low Latency and Low Cost
• Interleaved Group Convolutions for Deep Neural Networks
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Data Distillation: Towards Omni-Supervised Learning
• Knowledge Adaptation: Teaching to Adapt
• FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition
• DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classification
• Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision
• Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN
• Knowledge Projection for Deep Neural Networks
• Knowledge distillation using unlabeled mismatched images
• Knowledge Transfer Pre-training
• Age Group and Gender Estimation in the Wild with Deep RoR Architecture
• Feature Incay for Representation Regularization
• Modularized Morphing of Neural Networks
• Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification
• Share your Model instead of your Data: Privacy Preserving Mimic Learning for Ranking
• Generative Knowledge Transfer for Neural Language Models
• Multiscale Hierarchical Convolutional Networks
• On architectural choices in deep learning: From network structure to gradient convergence and parameter estimation
• Sequential Dynamic Decision Making with Deep Neural Nets on a Test-Time Budget
• Hardware-Software Codesign of Accurate, Multiplier-free Deep Neural Networks