Neighbourhood Distillation: On the benefits of non end-to-end distillation

Abstract

End-to-end training with back propagation is the standard method for training deep neural networks. However, as networks become deeper and bigger, end-to-end training becomes more challenging: highly non-convex models gets stuck easily in local optima, gradients signals are prone to vanish or explode during back-propagation, training requires computational resources and time. In this work, we propose to break away from the end-to-end paradigm in the context of Knowledge Distillation. Instead of distilling a model end-to-end, we propose to split it into smaller sub-networks - also called neighbourhoods - that are then trained independently. We empirically show that distilling networks in a non end-to-end fashion can be beneficial in a diverse range of use cases. First, we show that it speeds up Knowledge Distillation by exploiting parallelism and training on smaller networks. Second, we show that independently distilled neighbourhoods may be efficiently re-used for Neural Architecture Search. Finally, because smaller networks model simpler functions, we show that they are easier to train with synthetic data than their deeper counterparts.

arXiv comment16 pages, 8 figures. Update acknowledgements and fix typos

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• FitNets: Hints for Thin Deep Nets
• Bidirectional Attention Flow for Machine Comprehension
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Data-Free Knowledge Distillation for Deep Neural Networks
• Progressive Neural Architecture Search
• Training Neural Networks with Local Error Signals
• Are All Layers Created Equal?
• Moonshine: Distilling with Cheap Convolutions
• Knowledge Distillation via Route Constrained Optimization
• Dream Distillation: A Data-Independent Model Compression Framework
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• Greedy Layerwise Learning Can Scale to ImageNet
• Knowledge Distillation from Internal Representations
• Self-Knowledge Distillation in Natural Language Processing
• Knowledge Projection for Deep Neural Networks
• Towards Oracle Knowledge Distillation with Neural Architecture Search