Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students

Abstract

We focus on the problem of training a deep neural network in generations. The flowchart is that, in order to optimize the target network (student), another network (teacher) with the same architecture is first trained, and used to provide part of supervision signals in the next stage. While this strategy leads to a higher accuracy, many aspects (e.g., why teacher-student optimization helps) still need further explorations. This paper studies this problem from a perspective of controlling the strictness in training the teacher network. Existing approaches mostly used a hard distribution (e.g., one-hot vectors) in training, leading to a strict teacher which itself has a high accuracy, but we argue that the teacher needs to be more tolerant, although this often implies a lower accuracy. The implementation is very easy, with merely an extra loss term added to the teacher network, facilitating a few secondary classes to emerge and complement to the primary class. Consequently, the teacher provides a milder supervision signal (a less peaked distribution), and makes it possible for the student to learn from inter-class similarity and potentially lower the risk of over-fitting. Experiments are performed on standard image classification tasks (CIFAR100 and ILSVRC2012). Although the teacher network behaves less powerful, the students show a persistent ability growth and eventually achieve higher classification accuracies than other competitors. Model ensemble and transfer feature extraction also verify the effectiveness of our approach.

arXiv comment9 pages, 2 figures, major changes beyond v1

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• FitNets: Hints for Thin Deep Nets
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Stacked Hourglass Networks for Human Pose Estimation
• Born Again Neural Networks
• Shake-Shake regularization
• Holistically-Nested Edge Detection
• Net2Net: Accelerating Learning via Knowledge Transfer
• Deep Mutual Learning
• A hierarchical loss and its problems when classifying non-hierarchically

Cited by in corpus (27)

• Knowledge Distillation: A Survey
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Bridging the Modality Gap for Speech-to-Text Translation
• End-to-End Speech Translation with Knowledge Distillation
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Knowledge Distillation for End-to-End Person Search
• KDk: A Defense Mechanism Against Label Inference Attacks in Vertical Federated Learning
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
• Rethinking Normalization and Elimination Singularity in Neural Networks
• MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
• Thief, Beware of What Get You There: Towards Understanding Model Extraction Attack
• Towards Practical Lipreading with Distilled and Efficient Models
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• One-bit Supervision for Image Classification
• A Generalized and Robust Method Towards Practical Gaze Estimation on Smart Phone
• Knowledge Distillation via Instance-level Sequence Learning
• Student Network Learning via Evolutionary Knowledge Distillation
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• A Study of Non-autoregressive Model for Sequence Generation
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Differential Replication in Machine Learning
• Curvature: A signature for Action Recognition in Video Sequences
• Knowledge Distillation By Sparse Representation Matching