Large scale distributed neural network training through online distillation

Abstract

Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings. In this paper we explore a variant of distillation which is relatively straightforward to use as it does not require a complicated multi-stage setup or many new hyperparameters. Our first claim is that online distillation enables us to use extra parallelism to fit very large datasets about twice as fast. Crucially, we can still speed up training even after we have already reached the point at which additional parallelism provides no benefit for synchronous or asynchronous stochastic gradient descent. Two neural networks trained on disjoint subsets of the data can share knowledge by encouraging each model to agree with the predictions the other model would have made. These predictions can come from a stale version of the other model so they can be safely computed using weights that only rarely get transmitted. Our second claim is that online distillation is a cost-effective way to make the exact predictions of a model dramatically more reproducible. We support our claims using experiments on the Criteo Display Ad Challenge dataset, ImageNet, and the largest to-date dataset used for neural language modeling, containing $6\times 10^{11}$ tokens and based on the Common Crawl repository of web data.

arXiv commentClarify that implementations should use available parallelism in pseudo-code

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Revisiting Distributed Synchronous SGD
• Layer Normalization
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Deep Mutual Learning

Cited by in corpus (63)

• Knowledge Distillation: A Survey
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Knowledge Distillation by On-the-Fly Native Ensemble
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• Understanding and Improving Knowledge Distillation
• MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Bridging the Modality Gap for Speech-to-Text Translation
• End-to-End Speech Translation with Knowledge Distillation
• Hydra: Preserving Ensemble Diversity for Model Distillation
• Feature Fusion for Online Mutual Knowledge Distillation
• Distilling On-Device Intelligence at the Network Edge
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Distilling Object Detectors via Decoupled Features
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• Opening the black box of deep learning
• LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression
• Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
• Multiple Expert Brainstorming for Domain Adaptive Person Re-identification
• Learning Distilled Collaboration Graph for Multi-Agent Perception
• Anytime Inference with Distilled Hierarchical Neural Ensembles
• Revisiting Knowledge Distillation via Label Smoothing Regularization
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Heterogeneous Data-Aware Federated Learning
• Smooth activations and reproducibility in deep networks
• Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation
• ClusterFit: Improving Generalization of Visual Representations
• Anti-Distillation: Improving reproducibility of deep networks
• Small Towers Make Big Differences
• Federated Action Recognition on Heterogeneous Embedded Devices
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• Multilinear Compressive Learning with Prior Knowledge
• Online Knowledge Distillation via Multi-branch Diversity Enhancement
• Online Knowledge Distillation with Diverse Peers
• Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling
• Collaborative Deep Learning Across Multiple Data Centers
• Word Embedding based on Low-Rank Doubly Stochastic Matrix Decomposition
• Correlation Congruence for Knowledge Distillation
• Synthesizing Irreproducibility in Deep Networks
• Student Network Learning via Evolutionary Knowledge Distillation
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
• Curie: Policy-based Secure Data Exchange
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients
• Churn Reduction via Distillation
• Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack
• Prior knowledge distillation based on financial time series
• A Low Complexity Decentralized Neural Net with Centralized Equivalence using Layer-wise Learning
• The Scalability for Parallel Machine Learning Training Algorithm: Dataset Matters
• MOD: A Deep Mixture Model with Online Knowledge Distillation for Large Scale Video Temporal Concept Localization
• Locally Adaptive Label Smoothing for Predictive Churn
• A Study of Non-autoregressive Model for Sequence Generation
• Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
• Knowledge Transfer Graph for Deep Collaborative Learning
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
• Locally Linear Region Knowledge Distillation
• Dual-Teacher: Integrating Intra-domain and Inter-domain Teachers for Annotation-efficient Cardiac Segmentation
• Diversified Mutual Learning for Deep Metric Learning
• A Generalized Meta-loss function for regression and classification using privileged information
• Canoe : A System for Collaborative Learning for Neural Nets