Knowledge Distillation by On-the-Fly Native Ensemble

Abstract

Knowledge distillation is effective to train small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a highcapacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) strategy for one-stage online distillation. Specifically, ONE trains only a single multi-branch network while simultaneously establishing a strong teacher on-the- fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.

arXiv commentTo appear in NIPS2018

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• Deep Networks with Stochastic Depth
• Large scale distributed neural network training through online distillation
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Deep Mutual Learning
• Person Search by Multi-Scale Matching

Cited by in corpus (48)

• Knowledge Distillation: A Survey
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Knowledge Distillation in Deep Learning and its Applications
• Self-Distillation Amplifies Regularization in Hilbert Space
• Feature-map-level Online Adversarial Knowledge Distillation
• Contrastive Representation Distillation
• ResKD: Residual-Guided Knowledge Distillation
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• Learning Student-Friendly Teacher Networks for Knowledge Distillation
• Online Ensemble Model Compression using Knowledge Distillation
• Distilling Knowledge by Mimicking Features
• Hydra: Preserving Ensemble Diversity for Model Distillation
• Feature Fusion for Online Mutual Knowledge Distillation
• MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution
• FEED: Feature-level Ensemble for Knowledge Distillation
• Adaptive Regularization of Labels
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• Improved Techniques for Training Adaptive Deep Networks
• Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
• Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification
• Densely Guided Knowledge Distillation using Multiple Teacher Assistants
• EnsembleNet: End-to-End Optimization of Multi-headed Models
• An Embarrassingly Simple Approach for Knowledge Distillation
• PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors
• Patch Slimming for Efficient Vision Transformers
• Knowledge Distillation via Weighted Ensemble of Teaching Assistants
• PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
• Anti-Distillation: Improving reproducibility of deep networks
• Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels
• NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification
• Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation
• Fast Human Pose Estimation
• Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates
• Locally Adaptive Label Smoothing for Predictive Churn
• Multi-view Contrastive Learning for Online Knowledge Distillation
• Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement
• Knowledge Transfer Graph for Deep Collaborative Learning
• Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
• Resolution Switchable Networks for Runtime Efficient Image Recognition
• Semi-Online Knowledge Distillation
• A Survey on Green Deep Learning
• Locally Linear Region Knowledge Distillation
• Self-Referenced Deep Learning
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer
• Collaborative Group Learning