Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer

Abstract

Deep neural network architectures have attained remarkable improvements in scene understanding tasks. Utilizing an efficient model is one of the most important constraints for limited-resource devices. Recently, several compression methods have been proposed to diminish the heavy computational burden and memory consumption. Among them, the pruning and quantizing methods exhibit a critical drop in performances by compressing the model parameters. While the knowledge distillation methods improve the performance of compact models by focusing on training lightweight networks with the supervision of cumbersome networks. In the proposed method, the knowledge distillation has been performed within the network by constructing multiple branches over the primary stream of the model, known as the self-distillation method. Therefore, the ensemble of sub-neural network models has been proposed to transfer the knowledge among themselves with the knowledge distillation policies as well as an adversarial learning strategy. Hence, The proposed ensemble of sub-models is trained against a discriminator model adversarially. Besides, their knowledge is transferred within the ensemble by four different loss functions. The proposed method has been devoted to both lightweight image classification and encoder-decoder architectures to boost the performance of small and compact models without incurring extra computational overhead at the inference process. Extensive experimental results on the main challenging datasets show that the proposed network outperforms the primary model in terms of accuracy at the same number of parameters and computational cost. The obtained results show that the proposed model has achieved significant improvement over earlier ideas of self-distillation methods. The effectiveness of the proposed models has also been illustrated in the encoder-decoder model.

arXiv comment11 pages, 4 figures

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Knowledge Distillation: A Survey
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• A Survey of Deep Learning Techniques for Autonomous Driving
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Neural Networks with the Hashing Trick
• Knowledge Distillation by On-the-Fly Native Ensemble
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• Improving Fast Segmentation With Teacher-student Learning
• Lightweight Residual Densely Connected Convolutional Neural Network
• MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks
• KTAN: Knowledge Transfer Adversarial Network
• Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images
• Distilling Pixel-Wise Feature Similarities for Semantic Segmentation