Like What You Like: Knowledge Distill via Neuron Selectivity Transfer

Abstract

Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• Deep Domain Confusion: Maximizing for Domain Invariance
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Adversarial Discriminative Domain Adaptation
• Aggregated Residual Transformations for Deep Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• Demystifying Neural Style Transfer

Cited by in corpus (87)

• Knowledge Distillation: A Survey
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
• Incremental Object Detection via Meta-Learning
• Self-Distillation Amplifies Regularization in Hilbert Space
• SEED: Self-supervised Distillation For Visual Representation
• Contrastive Representation Distillation
• ResKD: Residual-Guided Knowledge Distillation
• Improving Fast Segmentation With Teacher-student Learning
• Small Sample Learning in Big Data Era
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Self-Distillation as Instance-Specific Label Smoothing
• Relational Knowledge Distillation
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation
• Variational Information Distillation for Knowledge Transfer
• TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
• FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models
• Kernel Based Progressive Distillation for Adder Neural Networks
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
• Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining
• FEED: Feature-level Ensemble for Knowledge Distillation
• DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer
• Distilling Knowledge from Graph Convolutional Networks
• Residual Knowledge Distillation
• Knowledge Adaptation for Efficient Semantic Segmentation
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Few Sample Knowledge Distillation for Efficient Network Compression
• Knowledge distillation via adaptive instance normalization
• Amalgamating Knowledge towards Comprehensive Classification
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• Attention-Guided Answer Distillation for Machine Reading Comprehension
• An Embarrassingly Simple Approach for Knowledge Distillation
• Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
• Multi-label Contrastive Predictive Coding
• Boosting Light-Weight Depth Estimation Via Knowledge Distillation
• Adaptive Instance Distillation for Object Detection in Autonomous Driving
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• CNN-based RGB-D Salient Object Detection: Learn, Select and Fuse
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision
• PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
• A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation
• General Instance Distillation for Object Detection
• Learning Student Networks via Feature Embedding
• A Convolutional Neural Network-Based Low Complexity Filter
• Streaming egocentric action anticipation: An evaluation scheme and approach
• Weakly Supervised 3D Object Detection from Point Clouds
• MarginDistillation: distillation for margin-based softmax
• AlphaNet: Improved Training of Supernets with Alpha-Divergence
• Semantically-Conditioned Negative Samples for Efficient Contrastive Learning
• Bidirectional Knowledge Reconfiguration for Lightweight Point Cloud Analysis
• Triplet Distillation for Deep Face Recognition
• Representation Transfer by Optimal Transport
• LNPT: Label-free Network Pruning and Training
• Improving Face Recognition from Hard Samples via Distribution Distillation Loss
• Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
• Revisiting Knowledge Distillation: An Inheritance and Exploration Framework
• Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices
• Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation
• Efficient Crowd Counting via Structured Knowledge Transfer
• Information Theoretic Representation Distillation
• Distilling Pixel-Wise Feature Similarities for Semantic Segmentation
• A Fast Knowledge Distillation Framework for Visual Recognition
• Matching Guided Distillation
• Learning from a Lightweight Teacher for Efficient Knowledge Distillation
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• Similarity Transfer for Knowledge Distillation
• Training a Binary Weight Object Detector by Knowledge Transfer for Autonomous Driving
• Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation
• RDPD: Rich Data Helps Poor Data via Imitation
• Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
• Complementary Relation Contrastive Distillation
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• Locally Linear Region Knowledge Distillation
• Multi-granularity for knowledge distillation
• A Studious Approach to Semi-Supervised Learning
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
• A Survey on Green Deep Learning
• On the Efficiency of Subclass Knowledge Distillation in Classification Tasks
• Self-Supervised Adaptation for Video Super-Resolution
• Diversified Mutual Learning for Deep Metric Learning
• CompConv: A Compact Convolution Module for Efficient Feature Learning