Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement

Abstract

Generic Image recognition is a fundamental and fairly important visual problem in computer vision. One of the major challenges of this task lies in the fact that single image usually has multiple objects inside while the labels are still one-hot, another one is noisy and sometimes missing labels when annotated by humans. In this paper, we focus on tackling these challenges accompanying with two different image recognition problems: multi-model ensemble and noisy data recognition with a unified framework. As is well-known, usually the best performing deep neural models are ensembles of multiple base-level networks, as it can mitigate the variation or noise containing in the dataset. Unfortunately, the space required to store these many networks, and the time required to execute them at runtime, prohibit their use in applications where test sets are large (e.g., ImageNet). In this paper, we present a method for compressing large, complex trained ensembles into a single network, where the knowledge from a variety of trained deep neural networks (DNNs) is distilled and transferred to a single DNN. In order to distill diverse knowledge from different trained (teacher) models, we propose to use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously. Extensive experiments on CIFAR-10/100, SVHN, ImageNet and iMaterialist Challenge Dataset demonstrate the effectiveness of our MEAL method. On ImageNet, our ResNet-50 based MEAL achieves top-1/5 21.79%/5.99% val error, which outperforms the original model by 2.06%/1.14%. On iMaterialist Challenge Dataset, our MEAL obtains a remarkable improvement of top-3 1.15% (official evaluation metric) on a strong baseline model of ResNet-101.

This is an extended version of our previous conference paper arXiv:1812.02425

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Toward Multimodal Image-to-Image Translation
• Learning to Reweight Examples for Robust Deep Learning
• Classification with Noisy Labels by Importance Reweighting
• Training Convolutional Networks with Noisy Labels
• Knowledge Distillation by On-the-Fly Native Ensemble
• Shake-Shake regularization
• On Convergence and Stability of GANs
• Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
• Label Refinery: Improving ImageNet Classification through Label Progression
• Swapout: Learning an ensemble of deep architectures
• Snapshot Ensembles: Train 1, get M for free
• Moonshine: Distilling with Cheap Convolutions