Regularizing Class-wise Predictions via Self-knowledge Distillation
arXiv:2003.13964
Abstract
Deep neural networks with millions of parameters may suffer from poor generalization due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label during training. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network (i.e., a self-knowledge distillation) by forcing it to produce more meaningful and consistent predictions in a class-wise manner. Consequently, it mitigates overconfident predictions and reduces intra-class variations. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve not only the generalization ability but also the calibration performance of modern convolutional neural networks.
Accepted to CVPR 2020. Code is available at https://github.com/alinlab/cs-kd
References in corpus (16)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Improved Regularization of Convolutional Neural Networks with Cutout
- Unsupervised Data Augmentation for Consistency Training
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- AutoAugment: Learning Augmentation Policies from Data
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Learning with Pseudo-Ensembles
- Recent Advance in Content-based Image Retrieval: A Literature Survey
- Label Refinery: Improving ImageNet Classification through Label Progression
- Knowledge Transfer with Jacobian Matching
- Range Loss for Deep Face Recognition with Long-tail
- Maximum-Entropy Fine-Grained Classification
- Virtual Class Enhanced Discriminative Embedding Learning