Deep Mutual Learning

Abstract

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, that is better suited to low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy where, rather than one way transfer between a static pre-defined teacher and a student, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on CIFAR-100 recognition and Market-1501 person re-identification benchmarks. Surprisingly, it is revealed that no prior powerful teacher network is necessary -- mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.

10 pages, 4 figures

References in corpus (4)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Do Deep Nets Really Need to be Deep?

Cited by in corpus (29)

• AlignedReID: Surpassing Human-Level Performance in Person Re-Identification
• Knowledge Distillation by On-the-Fly Native Ensemble
• Large scale distributed neural network training through online distillation
• Margin Sample Mining Loss: A Deep Learning Based Method for Person Re-identification
• Re-ID done right: towards good practices for person re-identification
• Dual Attention Matching Network for Context-Aware Feature Sequence based Person Re-Identification
• Discriminative Feature Learning with Foreground Attention for Person Re-Identification
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Distilling Policy Distillation
• Knowledge Distillation via Route Constrained Optimization
• Mix&Match - Agent Curricula for Reinforcement Learning
• FEED: Feature-level Ensemble for Knowledge Distillation
• Towards Effective Low-bitwidth Convolutional Neural Networks
• Few Sample Knowledge Distillation for Efficient Network Compression
• Exchangeable deep neural networks for set-to-set matching and learning
• Intra-Inter Camera Similarity for Unsupervised Person Re-Identification
• Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
• NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification
• Backbone Can Not be Trained at Once: Rolling Back to Pre-trained Network for Person Re-Identification
• Simple Distillation Baselines for Improving Small Self-supervised Models
• Search to Distill: Pearls are Everywhere but not the Eyes
• In Defense of the Classification Loss for Person Re-Identification
• MOD: A Deep Mixture Model with Online Knowledge Distillation for Large Scale Video Temporal Concept Localization
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Improving Route Choice Models by Incorporating Contextual Factors via Knowledge Distillation
• Virtual CNN Branching: Efficient Feature Ensemble for Person Re-Identification
• Self-Distilled Self-Supervised Representation Learning
• ISyNet: Convolutional Neural Networks design for AI accelerator
• GenURL: A General Framework for Unsupervised Representation Learning