A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Abstract

The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit such analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz., mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons for the success of the heuristics. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed to the deeper layers.

arXiv commentWe use empirical tools of mode connectivity and SVCCA to investigate neural network training heuristics of learning rate restarts, warmup and knowledge distillation. arXiv admin note: text overlap with arXiv:1806.06977

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Visualizing the Loss Landscape of Neural Nets
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Born Again Neural Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Averaging Weights Leads to Wider Optima and Better Generalization
• Three Factors Influencing Minima in SGD
• Essentially No Barriers in Neural Network Energy Landscape
• Convergent Learning: Do different neural networks learn the same representations?
• Theory II: Landscape of the Empirical Risk in Deep Learning
• Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation

Cited by in corpus (21)

• Transfusion: Understanding Transfer Learning for Medical Imaging
• Facial Emotion Recognition: State of the Art Performance on FER2013
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• SCPM-Net: An Anchor-free 3D Lung Nodule Detection Network using Sphere Representation and Center Points Matching
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Breast Cancer Diagnosis in Two-View Mammography Using End-to-End Trained EfficientNet-Based Convolutional Network
• Dive into Deep Learning
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
• An Adaptive and Momental Bound Method for Stochastic Learning
• On the adequacy of untuned warmup for adaptive optimization
• Physics-Assisted Reduced-Order Modeling for Identifying Dominant Features of Transonic Buffet
• Disentangling Adaptive Gradient Methods from Learning Rates
• Revisiting Knowledge Distillation for Object Detection
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• Large Batch Training Does Not Need Warmup
• Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums
• Verifiability and Predictability: Interpreting Utilities of Network Architectures for Point Cloud Processing
• Zero Training Overhead Portfolios for Learning to Solve Combinatorial Problems
• An Investigation of Traffic Density Changes inside Wuhan during the COVID-19 Epidemic with GF-2 Time-Series Images