Hydra: Preserving Ensemble Diversity for Model Distillation

Abstract

Ensembles of models have been empirically shown to improve predictive performance and to yield robust measures of uncertainty. However, they are expensive in computation and memory. Therefore, recent research has focused on distilling ensembles into a single compact model, reducing the computational and memory burden of the ensemble while trying to preserve its predictive behavior. Most existing distillation formulations summarize the ensemble by capturing its average predictions. As a result, the diversity of the ensemble predictions, stemming from each member, is lost. Thus, the distilled model cannot provide a measure of uncertainty comparable to that of the original ensemble. To retain more faithfully the diversity of the ensemble, we propose a distillation method based on a single multi-headed neural network, which we refer to as Hydra. The shared body network learns a joint feature representation that enables each head to capture the predictive behavior of each ensemble member. We demonstrate that with a slight increase in parameter count, Hydra improves distillation performance on classification and regression settings while capturing the uncertainty behavior of the original ensemble over both in-domain and out-of-distribution tasks.

Accepted to ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Knowledge Distillation by On-the-Fly Native Ensemble
• Bayesian Dark Knowledge
• Unifying distillation and privileged information
• Large scale distributed neural network training through online distillation
• Towards Understanding Knowledge Distillation
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation

Cited by in corpus (16)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Self-Distillation Amplifies Regularization in Hilbert Space
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Training independent subnetworks for robust prediction
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Twin Neural Network Regression
• Regression Prior Networks
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Diversity Matters When Learning From Ensembles
• Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets
• Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems
• Improving robustness and calibration in ensembles with diversity regularization
• Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
• Uncertainty Surrogates for Deep Learning
• Deep Classifiers with Label Noise Modeling and Distance Awareness