Born Again Neural Networks

Abstract

Knowledge Distillation (KD) consists of transferring “knowledge” from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the student’s compactness, without sacrificing too much performance. We study KD from a new perspective: rather than compressing models, we train students parameterized identically to their teachers. Surprisingly, these Born-Again Networks (BANs), outperform their teachers significantly, both on computer vision and language modeling tasks. Our experiments with BANs based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10 (3.5%) and CIFAR-100 (15.5%) datasets, by validation error. Additional experiments explore two distillation objectives: (i) Confidence-Weighted by Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP). Both methods elucidate the essential components of KD, demonstrating the effect of the teacher outputs on both predicted and non-predicted classes.

arXiv commentPublished @ICML 2018

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• Improved Regularization of Convolutional Neural Networks with Cutout
• Recurrent Neural Network Regularization
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Do Deep Nets Really Need to be Deep?
• Character-Aware Neural Language Models
• Regularizing and Optimizing LSTM Language Models
• Aggregated Residual Transformations for Deep Neural Networks
• Shake-Shake regularization
• Distilling a Neural Network Into a Soft Decision Tree
• Continual Learning with Deep Generative Replay
• ShakeDrop Regularization for Deep Residual Learning
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Policy Distillation
• Active Long Term Memory Networks
• Deep Pyramidal Residual Networks with Separated Stochastic Depth

Cited by in corpus (89)

• Knowledge Distillation: A Survey
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Distilled Siamese Networks for Visual Tracking
• Knowledge Distillation in Deep Learning and its Applications
• Scene Text Detection and Recognition: The Deep Learning Era
• Data-Free Adversarial Distillation
• Understanding and Improving Knowledge Distillation
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles -- Extended Version
• Training Strategies for Improved Lip-reading
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Geo-Localization via Ground-to-Satellite Cross-View Image Retrieval
• Synergic Adversarial Label Learning for Grading Retinal Diseases via Knowledge Distillation and Multi-task Learning
• Technical report on Conversational Question Answering
• Feature Fusion for Online Mutual Knowledge Distillation
• Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• MoËT: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning
• Born-Again Tree Ensembles
• Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data
• FS-BAN: Born-Again Networks for Domain Generalization Few-Shot Classification
• FEED: Feature-level Ensemble for Knowledge Distillation
• Self-Knowledge Distillation in Natural Language Processing
• Semi-Supervised Sequence Modeling with Cross-View Training
• Role-Wise Data Augmentation for Knowledge Distillation
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• Explaining Knowledge Distillation by Quantifying the Knowledge
• Revisiting Knowledge Distillation via Label Smoothing Regularization
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
• Learning Efficient Detector with Semi-supervised Adaptive Distillation
• Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
• Self-supervising Fine-grained Region Similarities for Large-scale Image Localization
• AdaNet: A Scalable and Flexible Framework for Automatically Learning Ensembles
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• Combinets: Creativity via Recombination of Neural Networks
• Improving Multi-Person Pose Estimation using Label Correction
• Ensemble Knowledge Distillation for Learning Improved and Efficient Networks
• DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognition
• Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
• Deep Serial Number: Computational Watermarking for DNN Intellectual Property Protection
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
• Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework
• Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
• Secost: Sequential co-supervision for large scale weakly labeled audio event detection
• The State of Knowledge Distillation for Classification
• EDF: Ensemble, Distill, and Fuse for Easy Video Labeling
• Making CNNs for Video Parsing Accessible
• ClusterFit: Improving Generalization of Visual Representations
• Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!
• MarginDistillation: distillation for margin-based softmax
• Explaining Neural Networks Semantically and Quantitatively
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Unifying Specialist Image Embedding into Universal Image Embedding
• Improving Neural Topic Models with Wasserstein Knowledge Distillation
• One-bit Supervision for Image Classification
• Triplet Distillation for Deep Face Recognition
• On Machine Learning and Structure for Mobile Robots
• Online Knowledge Distillation via Multi-branch Diversity Enhancement
• ADD: Augmented Disentanglement Distillation Framework for Improving Stock Trend Forecasting
• Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
• Knowledge Distillation via Instance-level Sequence Learning
• Linking Common Vulnerabilities and Exposures to the MITRE ATT&CK Framework: A Self-Distillation Approach
• Student Network Learning via Evolutionary Knowledge Distillation
• Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling
• Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
• Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation
• Efficient Action Recognition Using Confidence Distillation
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• Label Augmentation via Time-based Knowledge Distillation for Financial Anomaly Detection
• Learning Realistic Patterns from Unrealistic Stimuli: Generalization and Data Anonymization
• Recurrent Distillation based Crowd Counting
• Long Short-Term Sample Distillation
• Dual Correction Strategy for Ranking Distillation in Top-N Recommender System
• Weight Distillation: Transferring the Knowledge in Neural Network Parameters
• DNN Quantization with Attention
• Supervised Domain Enablement Attention for Personalized Domain Classification
• Learning from Matured Dumb Teacher for Fine Generalization