Knowledge Squeezed Adversarial Network Compression

Abstract

Deep network compression has been achieved notable progress via knowledge distillation, where a teacher-student learning manner is adopted by using predetermined loss. Recently, more focuses have been transferred to employ the adversarial training to minimize the discrepancy between distributions of output from two networks. However, they always emphasize on result-oriented learning while neglecting the scheme of process-oriented learning, leading to the loss of rich information contained in the whole network pipeline. Inspired by the assumption that, the small network can not perfectly mimic a large one due to the huge gap of network scale, we propose a knowledge transfer method, involving effective intermediate supervision, under the adversarial training framework to learn the student network. To achieve powerful but highly compact intermediate information representation, the squeezed knowledge is realized by task-driven attention mechanism. Then, the transferred knowledge from teacher network could accommodate the size of student network. As a result, the proposed method integrates merits from both process-oriented and result-oriented learning. Extensive experimental results on three typical benchmark datasets, i.e., CIFAR-10, CIFAR-100, and ImageNet, demonstrate that our method achieves highly superior performances against other state-of-the-art methods.

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Recurrent Models of Visual Attention
• Ternary Weight Networks
• Trained Ternary Quantization
• Pruning Filters for Efficient ConvNets
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Convolutional neural networks with low-rank regularization
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Model compression via distillation and quantization
• Learn To Pay Attention
• Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net
• KTAN: Knowledge Transfer Adversarial Network
• Attention-Guided Answer Distillation for Machine Reading Comprehension

Cited by in corpus (6)

• Knowledge Distillation: A Survey
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
• Source-Free Domain Adaptation for Semantic Segmentation
• Towards Efficient Unconstrained Palmprint Recognition via Deep Distillation Hashing
• Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimization
• Class-dependent Compression of Deep Neural Networks