Knowledge Distillation: A Survey

Abstract

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due to its scalability to encode large-scale data and to maneuver billions of model parameters. However, it is a challenge to deploy these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded devices, not only because of the high computational complexity but also the large storage requirements. To this end, a variety of model compression and acceleration techniques have been developed. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model. It has received rapid increasing attention from the community. This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison and applications. Furthermore, challenges in knowledge distillation are briefly reviewed and comments on future research are discussed and forwarded.

It has been accepted for publication in International Journal of Computer Vision (2021)

References in corpus (156)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• FitNets: Hints for Thin Deep Nets
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• When Does Label Smoothing Help?
• Learning both Weights and Connections for Efficient Neural Networks
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Born Again Neural Networks
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Knowledge Distillation by On-the-Fly Native Ensemble
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Model compression via distillation and quantization
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Adaptive Multi-Teacher Multi-level Knowledge Distillation
• Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation
• Data-Free Knowledge Distillation for Deep Neural Networks
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Adversarially Robust Distillation
• Unifying distillation and privileged information
• Net2Net: Accelerating Learning via Knowledge Transfer
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Large scale distributed neural network training through online distillation
• Label Refinery: Improving ImageNet Classification through Label Progression
• Towards Understanding Knowledge Distillation
• Multilingual Neural Machine Translation with Knowledge Distillation
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Sequence-Level Knowledge Distillation
• Conditional Teacher-Student Learning
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Structured Transforms for Small-Footprint Deep Learning
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Self-Distillation Amplifies Regularization in Hilbert Space
• Learning Lightweight Lane Detection CNNs by Self Attention Distillation
• Moonshine: Distilling with Cheap Convolutions
• Understanding and Improving Knowledge Distillation
• Paraphrasing Complex Network: Network Compression via Factor Transfer
• Feature-map-level Online Adversarial Knowledge Distillation
• Comprehensive Attention Self-Distillation for Weakly-Supervised Object Detection
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Ensemble Distillation for Neural Machine Translation
• Compressing GANs using Knowledge Distillation
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Transferring Knowledge from a RNN to a DNN
• Self-Distillation as Instance-Specific Label Smoothing
• Relational Knowledge Distillation
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• QKD: Quantization-aware Knowledge Distillation
• Online Ensemble Model Compression using Knowledge Distillation
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• Knowledge Adaptation: Teaching to Adapt
• Object Relational Graph with Teacher-Recommended Learning for Video Captioning
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
• Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net
• Transferring Knowledge across Learning Processes
• Graph-based Knowledge Distillation by Multi-head Attention Network
• Knowledge Distillation via Route Constrained Optimization
• Variational Information Distillation for Knowledge Transfer
• Federated Knowledge Distillation
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• Doubly Convolutional Neural Networks
• Feature Fusion for Online Mutual Knowledge Distillation
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
• TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
• Regularizing Class-wise Predictions via Self-knowledge Distillation
• Neural Compatibility Modeling with Attentive Knowledge Distillation
• ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks
• Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
• Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
• Adaptive Regularization of Labels
• FEED: Feature-level Ensemble for Knowledge Distillation
• Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control
• Distilling Knowledge from Graph Convolutional Networks
• An Information-Theoretic View for Deep Learning
• In Teacher We Trust: Learning Compressed Models for Pedestrian Detection
• Self-Knowledge Distillation in Natural Language Processing
• Residual Knowledge Distillation
• Knowledge Adaptation for Efficient Semantic Segmentation
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
• Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation
• Flexible Dataset Distillation: Learn Labels Instead of Images
• Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
• DDFlow: Learning Optical Flow with Unlabeled Data Distillation
• Few Shot Network Compression via Cross Distillation
• Few Sample Knowledge Distillation for Efficient Network Compression
• Explaining Knowledge Distillation by Quantifying the Knowledge
• Knowledge distillation via adaptive instance normalization
• KTAN: Knowledge Transfer Adversarial Network
• Revisiting Knowledge Distillation via Label Smoothing Regularization
• Distilling Word Embeddings: An Encoding Approach
• Knowledge Distillation for End-to-End Person Search
• Compact Trilinear Interaction for Visual Question Answering
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Search for Better Students to Learn Distilled Knowledge
• Knowledge Squeezed Adversarial Network Compression
• Teacher-Student Compression with Generative Adversarial Networks
• Attention-Guided Answer Distillation for Machine Reading Comprehension
• Unpaired Multi-modal Segmentation via Knowledge Distillation
• Learning Deep Representations with Probabilistic Knowledge Transfer
• Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More
• Dual Policy Distillation
• Ensemble Knowledge Distillation for Learning Improved and Efficient Networks
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
• Dynamic Kernel Distillation for Efficient Pose Estimation in Videos
• Graph Representation Learning via Multi-task Knowledge Distillation
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• Inter-Region Affinity Distillation for Road Marking Segmentation
• Progressive Network Grafting for Few-Shot Knowledge Distillation
• Learning Metrics from Teachers: Compact Networks for Image Embedding
• Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer
• Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning
• Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
• Efficient Video Classification Using Fewer Frames
• Knowledge distillation for optimization of quantized deep neural networks
• Knowledge Transfer via Dense Cross-Layer Mutual-Distillation
• Discriminability Distillation in Group Representation Learning
• Diverse Knowledge Distillation for End-to-End Person Search
• Search to Distill: Pearls are Everywhere but not the Eyes
• Unifying Heterogeneous Classifiers with Distillation
• Deep Net Triage: Analyzing the Importance of Network Layers via Structural Compression
• Cross-modal knowledge distillation for action recognition
• Two-stage Image Classification Supervised by a Single Teacher Single Student Model
• Training convolutional neural networks with cheap convolutions and online distillation
• Peer Collaborative Learning for Online Knowledge Distillation
• Knowledge Distillation in Document Retrieval
• Knowledge Integration Networks for Action Recognition
• Matching Guided Distillation
• Reinforced Multi-Teacher Selection for Knowledge Distillation
• Defocus Blur Detection via Depth Distillation
• Prime-Aware Adaptive Distillation
• Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization
• Knowledge Transfer Graph for Deep Collaborative Learning
• Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation

Cited by in corpus (203)

• Structured Pruning for Deep Convolutional Neural Networks: A survey
• A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directions
• Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey
• A Comprehensive Survey of Dataset Distillation
• TrAISformer -- A Transformer Network with Sparse Augmented Data Representation and Cross Entropy Loss for AIS-based Vessel Trajectory Prediction
• Artificial Neural Networks for Photonic Applications: From Algorithms to Implementation
• Deep Long-Tailed Learning: A Survey
• Data Minimization for GDPR Compliance in Machine Learning Models
• Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation
• Learning from models beyond fine-tuning
• On Representation Knowledge Distillation for Graph Neural Networks
• ResKD: Residual-Guided Knowledge Distillation
• Unified Multi-Modal Image Synthesis for Missing Modality Imputation
• MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
• Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation
• Lightweight Pixel Difference Networks for Efficient Visual Representation Learning
• Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfer
• Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching
• Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Knowledge Distillation approach towards Melanoma Detection
• DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval
• Distilling Object Detectors with Feature Richness
• On-device Training: A First Overview on Existing Systems
• One-shot Federated Learning without Server-side Training
• Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking
• Introspective Distillation for Robust Question Answering
• Greening Large Language Models of Code
• Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation
• Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
• Trustable Co-label Learning from Multiple Noisy Annotators
• Architectural Vision for Quantum Computing in the Edge-Cloud Continuum
• Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillation
• AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization
• Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences
• Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model
• MoËT: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning
• Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation
• Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression
• Edge-aware Feature Aggregation Network for Polyp Segmentation
• Similarity of Neural Network Models: A Survey of Functional and Representational Measures
• Neural-IMLS: Self-supervised Implicit Moving Least-Squares Network for Surface Reconstruction
• Exploring Social Media for Early Detection of Depression in COVID-19 Patients
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• An Empirical Study of Challenges in Machine Learning Asset Management
• A Dimensionality Reduction Approach for Convolutional Neural Networks
• Theoretical research on generative diffusion models: an overview
• Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
• Towards On-Board Panoptic Segmentation of Multispectral Satellite Images
• Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
• Unveiling the frontiers of deep learning: innovations shaping diverse domains
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
• Self-Distribution Binary Neural Networks
• RankFormer: Listwise Learning-to-Rank Using Listwide Labels
• Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation
• A Comprehensive Survey on Imbalanced Data Learning
• Synthetic data generation method for data-free knowledge distillation in regression neural networks
• DASS: Differentiable Architecture Search for Sparse neural networks
• SOLD: Sinhala Offensive Language Dataset
• KDk: A Defense Mechanism Against Label Inference Attacks in Vertical Federated Learning
• Maximizing Discrimination Capability of Knowledge Distillation with Energy Function
• Foundation Model-Based Apple Ripeness and Size Estimation for Selective Harvesting
• MECKD: Deep Learning-Based Fall Detection in Multilayer Mobile Edge Computing With Knowledge Distillation
• Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning
• PowerSkel: A Device-Free Framework Using CSI Signal for Human Skeleton Estimation in Power Station
• Achieve Fairness without Demographics for Dermatological Disease Diagnosis
• Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
• Deep learning in automated ultrasonic NDE -- developments, axioms and opportunities
• LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
• A methodology for training homomorphicencryption friendly neural networks
• Distilling a Deep Neural Network into a Takagi-Sugeno-Kang Fuzzy Inference System
• Soft Prompt Decoding for Multilingual Dense Retrieval
• TIE-KD: Teacher-Independent and Explainable Knowledge Distillation for Monocular Depth Estimation
• Dual-Branch Knowledge Distillation for Noise-Robust Synthetic Speech Detection
• X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation
• Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications
• Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• Deep Serial Number: Computational Watermarking for DNN Intellectual Property Protection
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messages
• Federated Learning on Non-IID Data: A Survey
• Reconciling Attribute and Structural Anomalies for Improved Graph Anomaly Detection
• Data Distillation for Text Classification
• Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
• Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data
• Leveraging Angular Distributions for Improved Knowledge Distillation
• ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
• Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference
• Exploring the Mutual Influence between Self-Supervised Single-Frame and Multi-Frame Depth Estimation
• MIME: Adapting a Single Neural Network for Multi-task Inference with Memory-efficient Dynamic Pruning
• Keyword localisation in untranscribed speech using visually grounded speech models
• Knowledge Distillation on Spatial-Temporal Graph Convolutional Network for Traffic Prediction
• Collaborative Teacher-Student Learning via Multiple Knowledge Transfer
• CORE-ReID: Comprehensive Optimization and Refinement through Ensemble fusion in Domain Adaptation for person re-identification
• Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data
• Towards Incremental Learning in Large Language Models: A Critical Review
• Adaptive Intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems
• Teachers Do More Than Teach: Compressing Image-to-Image Models
• Anti-Distillation: Improving reproducibility of deep networks
• Prototype Memory for Large-scale Face Representation Learning
• Foundation Models Knowledge Distillation For Battery Capacity Degradation Forecast
• Learning to Teach with Student Feedback
• Blockchain-assisted Demonstration Cloning for Multi-Agent Deep Reinforcement Learning
• ESAI: Efficient Split Artificial Intelligence via Early Exiting Using Neural Architecture Search
• Cyber-Physical Systems Security: A Comprehensive Review of Anomaly Detection Techniques
• LegoNet: Memory Footprint Reduction Through Block Weight Clustering
• How and When Adversarial Robustness Transfers in Knowledge Distillation?
• Students are the Best Teacher: Exit-Ensemble Distillation with Multi-Exits
• Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition
• Streaming egocentric action anticipation: An evaluation scheme and approach
• Compressing Neural Networks Using Tensor Networks with Exponentially Fewer Variational Parameters
• Knowledge Distillation in RNN-Attention Models for Early Prediction of Student Performance
• Bidirectional Knowledge Reconfiguration for Lightweight Point Cloud Analysis
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• ImageNet-21K Pretraining for the Masses
• Knowledge Distillation as Semiparametric Inference
• Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation
• FedBrain-Distill: Communication-Efficient Federated Brain Tumor Classification Using Ensemble Knowledge Distillation on Non-IID Data
• Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
• Improving Neural Topic Models with Wasserstein Knowledge Distillation
• MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
• Leveraging Knowledge Distillation for Efficient Deep Reinforcement Learning in Resource-Constrained Environments
• ALPINE: An adaptive language-agnostic pruning method for language models for code
• DKDL-Net: A Lightweight Bearing Fault Detection Model via Decoupled Knowledge Distillation and Low-Rank Adaptation Fine-tuning
• Scaling Up Quantization-Aware Neural Architecture Search for Efficient Deep Learning on the Edge
• Toward a Unified Framework for Debugging Concept-based Models
• Activation Sparsity Opportunities for Compressing General Large Language Models
• On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards
• I2CKD : Intra- and Inter-Class Knowledge Distillation for Semantic Segmentation
• ShiftKD: Benchmarking Knowledge Distillation under Distribution Shift
• Growing and Serving Large Open-domain Knowledge Graphs
• One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers
• FakeSwarm: Improving Fake News Detection with Swarming Characteristics
• Using Neighborhood Context to Improve Information Extraction from Visual Documents Captured on Mobile Phones
• Response-based Distillation for Incremental Object Detection
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• Ground Reaction Force Estimation via Time-aware Knowledge Distillation
• Linear Item-Item Model with Neural Knowledge for Session-based Recommendation
• Selective Knowledge Distillation for Neural Machine Translation
• Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition
• A Fast Knowledge Distillation Framework for Visual Recognition
• Real-Time Cell Sorting with Scalable In Situ FPGA-Accelerated Deep Learning
• A Systematic Review of ECG Arrhythmia Classification: Adherence to Standards, Fair Evaluation, and Embedded Feasibility
• RankDVQA-mini: Knowledge Distillation-Driven Deep Video Quality Assessment
• BeSound: Bluetooth-Based Position Estimation Enhancing with Cross-Modality Distillation
• Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detection
• Distributed Edge Inference: an Experimental Study on Multiview Detection
• Comparative Analysis of Transformer Models in Disaster Tweet Classification for Public Safety
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
• SciceVPR: Stable Cross-Image Correlation Enhanced Model for Visual Place Recognition
• A Survey on GAN Acceleration Using Memory Compression Technique
• Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
• Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks
• Complementary Relation Contrastive Distillation
• Semi-Online Knowledge Distillation
• LADSG: Label-Anonymized Distillation and Similar Gradient Substitution for Label Privacy in Vertical Federated Learning
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• EEG aided boosting of single-lead ECG based sleep staging with Deep Knowledge Distillation
• FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer
• Emotion recognition in talking-face videos using persistent entropy and neural networks
• GNN's Uncertainty Quantification using Self-Distillation
• KLiNQ: Knowledge Distillation-Assisted Lightweight Neural Network for Qubit Readout on FPGA
• Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins
• Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillation
• Offline RL With Resource Constrained Online Deployment
• How to Explain Neural Networks: an Approximation Perspective
• Selective Correlation Based Knowledge Distillation for Ground Reaction Force Estimation
• Student-Teacher Learning from Clean Inputs to Noisy Inputs
• Knowledge Distillation for mmWave Beam Prediction Using Sub-6 GHz Channels
• Domain-Agnostic Clustering with Self-Distillation
• Evolving Knowledge Distillation for Lightweight Neural Machine Translation
• Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation
• OBSeg: Accurate and Fast Instance Segmentation Framework Using Segmentation Foundation Models with Oriented Bounding Box Prompts
• Embedding Subspace Partitioning for Dynamic Multi-Objective Retrieval
• TinyML NLP Scheme for Semantic Wireless Sentiment Classification with Privacy Preservation
• Role of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data
• Reformulation is All You Need: Addressing Malicious Text Features in DNNs
• Learning Compatible Embeddings
• Knowledge Distillation for Intelligent Softwarized Networks: Advances and Open Challenges
• Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification
• Multi-granularity for knowledge distillation
• Pose-Guided Feature Learning with Knowledge Distillation for Occluded Person Re-Identification
• HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networks
• Knowledge Distillation Decision Tree for Unravelling Black-box Machine Learning Models
• EXACT: How to Train Your Accuracy
• BERMo: What can BERT learn from ELMo?
• Creating Simple, Interpretable Anomaly Detectors for New Physics in Jet Substructure
• Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models
• Single Snapshot Distillation for Phase Coded Mask Design in Phase Retrieval
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Annealing Knowledge Distillation
• Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression
• TAGLETS: A System for Automatic Semi-Supervised Learning with Auxiliary Data
• Enhancing deep learning models for time series classification via knowledge distillation
• On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks
• Intra-class Patch Swap for Self-Distillation
• Tips and Tricks for Webly-Supervised Fine-Grained Recognition: Learning from the WebFG 2020 Challenge
• LISTEN: Lightweight Industrial Sound-representable Transformer for Edge Notification
• Faster Molecular Dynamics with Neural Network Potentials via Distilled Multiple Time-Stepping and Non-Conservative Forces