Feature-map-level Online Adversarial Knowledge Distillation

Abstract

Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network's feature map distribution. We show that our method performs better than the conventional direct alignment method such as L1 and is more suitable for online distillation. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• Pruning Filters for Efficient ConvNets
• Knowledge Distillation by On-the-Fly Native Ensemble
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• Adversarial Training Methods for Semi-Supervised Text Classification
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• KTAN: Knowledge Transfer Adversarial Network

Cited by in corpus (20)

• Knowledge Distillation: A Survey
• Bringing AI To Edge: From Deep Learning's Perspective
• A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
• Knowledge Distillation in Deep Learning and its Applications
• Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation
• PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors
• Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation
• HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast Images
• Local-Selective Feature Distillation for Single Image Super-Resolution
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• ShadowTutor: Distributed Partial Distillation for Mobile Video DNN Inference
• CEKD:Cross Ensemble Knowledge Distillation for Augmented Fine-grained Data
• Student Network Learning via Evolutionary Knowledge Distillation
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Self-Distilled Self-Supervised Representation Learning
• Dual Discriminator Adversarial Distillation for Data-free Model Compression
• Improved Knowledge Distillation via Adversarial Collaboration
• Locally Linear Region Knowledge Distillation
• A Survey on Green Deep Learning