Feature Fusion for Online Mutual Knowledge Distillation

Abstract

We propose a learning framework named Feature Fusion Learning (FFL) that efficiently trains a powerful classifier through a fusion module which combines the feature maps generated from parallel neural networks. Specifically, we train a number of parallel neural networks as sub-networks, then we combine the feature maps from each sub-network using a fusion module to create a more meaningful feature map. The fused feature map is passed into the fused classifier for overall classification. Unlike existing feature fusion methods, in our framework, an ensemble of sub-network classifiers transfers its knowledge to the fused classifier and then the fused classifier delivers its knowledge back to each sub-network, mutually teaching one another in an online-knowledge distillation manner. This mutually teaching system not only improves the performance of the fused classifier but also obtains performance gain in each sub-network. Moreover, our model is more beneficial because different types of network can be used for each sub-network. We have performed a variety of experiments on multiple datasets such as CIFAR-10, CIFAR-100 and ImageNet and proved that our method is more effective than other alternative methods in terms of performance of both sub-networks and the fused classifier.

arXiv commentInternational Conference on Pattern Recognition

References in corpus (17)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• Temporal Ensembling for Semi-Supervised Learning
• Training Very Deep Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Born Again Neural Networks
• Knowledge Distillation by On-the-Fly Native Ensemble
• Large scale distributed neural network training through online distillation
• Collaborative Learning for Deep Neural Networks
• Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
• QKD: Quantization-aware Knowledge Distillation
• Position-based Scaled Gradient for Model Quantization and Pruning

Cited by in corpus (8)

• Knowledge Distillation: A Survey
• Knowledge Distillation in Deep Learning and its Applications
• QKD: Quantization-aware Knowledge Distillation
• An Ensemble of Knowledge Sharing Models for Dynamic Hand Gesture Recognition
• Knowledge Distillation via Weighted Ensemble of Teaching Assistants
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Prototype-based Personalized Pruning
• Decision Support for Video-based Detection of Flu Symptoms