EnsembleNet: End-to-End Optimization of Multi-headed Models

Abstract

Ensembling is a universally useful approach to boost the performance of machine learning models. However, individual models in an ensemble were traditionally trained independently in separate stages without information access about the overall ensemble. Many co-distillation approaches were proposed in order to treat model ensembling as first-class citizens. In this paper, we reveal a deeper connection between ensembling and distillation, and come up with a simpler yet more effective co-distillation architecture. On large-scale datasets including ImageNet, YouTube-8M, and Kinetics, we demonstrate a general procedure that can convert a single deep neural network to a multi-headed model that has not only a smaller size but also better performance. The model can be optimized end-to-end with our proposed co-distillation loss in a single stage without human intervention.

References in corpus (11)

• Popular Ensemble Methods: An Empirical Study
• The Kinetics Human Action Video Dataset
• SGDR: Stochastic Gradient Descent with Warm Restarts
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Knowledge Distillation by On-the-Fly Native Ensemble
• Some Improvements on Deep Convolutional Neural Network Based Image Classification
• Learnable pooling with Context Gating for video classification
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Large scale distributed neural network training through online distillation
• SlowFast Networks for Video Recognition
• Deep Mixture of Experts via Shallow Embedding

Cited by in corpus (4)

• On Power Laws in Deep Ensembles
• Feature Guided Training and Rotational Standardisation for the Morphological Classification of Radio Galaxies
• Group Ensemble: Learning an Ensemble of ConvNets in a single ConvNet
• Adversarially Robust Frame Sampling with Bounded Irregularities