Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks

Abstract

Deep neural models in recent years have been successful in almost every field, including extremely complex problem statements. However, these models are huge in size, with millions (and even billions) of parameters, thus demanding more heavy computation power and failing to be deployed on edge devices. Besides, the performance boost is highly dependent on redundant labeled data. To achieve faster speeds and to handle the problems caused by the lack of data, knowledge distillation (KD) has been proposed to transfer information learned from one model to another. KD is often characterized by the so-called `Student-Teacher' (S-T) learning framework and has been broadly applied in model compression and knowledge transfer. This paper is about KD and S-T learning, which are being actively studied in recent years. First, we aim to provide explanations of what KD is and how/why it works. Then, we provide a comprehensive survey on the recent progress of KD methods together with S-T frameworks typically for vision tasks. In general, we consider some fundamental questions that have been driving this research area and thoroughly generalize the research progress and technical details. Additionally, we systematically analyze the research status of KD in vision applications. Finally, we discuss the potentials and open challenges of existing methods and prospect the future directions of KD and S-T learning.

Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence(TPAMI),2021. Some references are updated in this version

References in corpus (2)

• Deceiving Image-to-Image Translation Networks for Autonomous Driving with Adversarial Perturbations
• Knowledge Squeezed Adversarial Network Compression

Cited by in corpus (23)

• Structured Pruning for Deep Convolutional Neural Networks: A survey
• Tiny Machine Learning: Progress and Futures
• FedICT: Federated Multi-task Distillation for Multi-access Edge Computing
• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• Vision-based Learning for Drones: A Survey
• Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
• Bayesian Inference of Initial Conditions from Non-Linear Cosmic Structures using Field-Level Emulators
• Complementary Calibration: Boosting General Continual Learning with Collaborative Distillation and Self-Supervision
• Optimizing Convolutional Neural Network Architecture
• Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
• FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing
• Synthetic data generation method for data-free knowledge distillation in regression neural networks
• Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling
• Medical Instrument Segmentation in 3D US by Hybrid Constrained Semi-Supervised Learning
• Supervised Learning in the Presence of Concept Drift: A modelling framework
• Cyber Attacks Prevention Towards Prosumer-based EV Charging Stations: An Edge-assisted Federated Prototype Knowledge Distillation Approach
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
• RankDVQA-mini: Knowledge Distillation-Driven Deep Video Quality Assessment
• Unsupervised Learning for Industrial Defect Detection: A Case Study on Shearographic Data
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• Multiple Teachers-Meticulous Student: A Domain Adaptive Meta-Knowledge Distillation Model for Medical Image Classification
• Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models