Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

Abstract

We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.

arXiv commentv2/V3 polishes writing

References in corpus (35)

• Distilling the Knowledge in a Neural Network
• Popular Ensemble Methods: An Empirical Study
• Wide Residual Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Born Again Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Knowledge Distillation by On-the-Fly Native Ensemble
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Recovery Guarantees for One-hidden-layer Neural Networks
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Learning One-hidden-layer Neural Networks with Landscape Design
• On the Convergence Rate of Training Recurrent Neural Networks
• Self-Distillation Amplifies Regularization in Hilbert Space
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• SGD Learns the Conjugate Kernel Class of the Network
• Diverse Neural Network Learns True Target Functions
• Ensemble Distillation for Neural Machine Translation
• Neural Kernels Without Tangents
• Theoretical properties of the global optimizer of two layer neural network
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Recovery Guarantee of Non-negative Matrix Factorization via Alternating Updates
• Provable Alternating Gradient Descent for Non-negative Matrix Factorization with Strong Correlations
• Learning Two Layer Rectified Neural Networks in Polynomial Time
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK

Cited by in corpus (37)

• R-Drop: Regularized Dropout for Neural Networks
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Knowledge Distillation in Deep Learning and its Applications
• Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classification
• Beyond Heart Murmur Detection: Automatic Murmur Grading from Phonocardiogram
• Photoacoustic image synthesis with generative adversarial networks
• Improving Multi-Modal Learning with Uni-Modal Teachers
• Easy Ensemble: Simple Deep Ensemble Learning for Sensor-Based Human Activity Recognition
• FS-BAN: Born-Again Networks for Domain Generalization Few-Shot Classification
• What is Next when Sequential Prediction Meets Implicitly Hard Interaction?
• Assessing Generalization of SGD via Disagreement
• BEBERT: Efficient and Robust Binary Ensemble BERT
• Variant Parallelism: Lightweight Deep Convolutional Models for Distributed Inference on IoT Devices
• Semantic Segmentation in Multiple Adverse Weather Conditions with Domain Knowledge Retention
• Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems
• Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
• Importance Sampling CAMs for Weakly-Supervised Segmentation
• Students are the Best Teacher: Exit-Ensemble Distillation with Multi-Exits
• Teacher's pet: understanding and mitigating biases in distillation
• DistillCSE: Distilled Contrastive Learning for Sentence Embeddings
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• A Closer Look at Codistillation for Distributed Training
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimization
• Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy
• OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction
• Making Method of Moments Great Again? -- How can GANs learn distributions
• sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging
• Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
• Explaining generalization in deep learning: progress and fundamental limits
• Deep Learning Theory through the Lens of Statistical Physics: A Topical Review
• Technical Report of Team GraphMIRAcles in the WikiKG90M-LSC Track of OGB-LSC @ KDD Cup 2021
• Combining Diverse Feature Priors
• Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods