Few Shot Network Compression via Cross Distillation

Abstract

Model compression has been widely adopted to obtain light-weighted deep neural networks. Most prevalent methods, however, require fine-tuning with sufficient training data to ensure accuracy, which could be challenged by privacy and security issues. As a compromise between privacy and performance, in this paper we investigate few shot network compression: given few samples per class, how can we effectively compress the network with negligible performance drop? The core challenge of few shot network compression lies in high estimation errors from the original network during inference, since the compressed network can easily over-fits on the few training instances. The estimation errors could propagate and accumulate layer-wisely and finally deteriorate the network output. To address the problem, we propose cross distillation, a novel layer-wise knowledge distillation approach. By interweaving hidden layers of teacher and student network, layer-wisely accumulated estimation errors can be effectively reduced.The proposed method offers a general framework compatible with prevalent network compression techniques such as pruning. Extensive experiments on benchmark datasets demonstrate that cross distillation can significantly improve the student network's accuracy when only a few training instances are available.

arXiv commentAAAI 2020

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• FitNets: Hints for Thin Deep Nets
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Rethinking the Value of Network Pruning
• Pruning Filters for Efficient ConvNets
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Channel Pruning for Accelerating Very Deep Neural Networks
• Data-Free Knowledge Distillation for Deep Neural Networks
• Scalable Methods for 8-bit Training of Neural Networks
• ProxQuant: Quantized Neural Networks via Proximal Operators
• Dream Distillation: A Data-Independent Model Compression Framework
• Accelerating Very Deep Convolutional Networks for Classification and Detection
• Few Sample Knowledge Distillation for Efficient Network Compression

Cited by in corpus (5)

• Knowledge Distillation: A Survey
• BinaryBERT: Pushing the Limit of BERT Quantization
• Black-Box Ripper: Copying black-box models using generative evolutionary algorithms
• Neural Network Pruning with Residual-Connections and Limited-Data
• Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimization