Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule

Abstract

Several papers argue that wide minima generalize better than narrow minima. In this paper, through detailed experiments that not only corroborate the generalization properties of wide minima, we also provide empirical evidence for a new hypothesis that the density of wide minima is likely lower than the density of narrow minima. Further, motivated by this hypothesis, we design a novel explore-exploit learning rate schedule. On a variety of image and natural language datasets, compared to their original hand-tuned learning rate baselines, we show that our explore-exploit schedule can result in either up to 0.84% higher absolute accuracy using the original training budget or up to 57% reduced training time while achieving the original reported accuracy. For example, we achieve state-of-the-art (SOTA) accuracy for IWSLT'14 (DE-EN) dataset by just modifying the learning rate schedule of a high performing model.

arXiv comment34 pages

References in corpus (25)

• Adam: A Method for Stochastic Optimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• On the Variance of the Adaptive Learning Rate and Beyond
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Three Factors Influencing Minima in SGD
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Essentially No Barriers in Neural Network Energy Landscape
• Measuring the Effects of Data Parallelism on Neural Network Training
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• An Empirical Model of Large-Batch Training
• Sharp Minima Can Generalize For Deep Nets
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Shaping the learning landscape in neural networks around wide flat minima
• Stronger generalization bounds for deep nets via a compression approach
• A Walk with SGD
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• Identifying Generalization Properties in Neural Networks
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Towards Understanding Generalization in Gradient-Based Meta-Learning
• Information Geometry of Orthogonal Initializations and Training

Cited by in corpus (5)

• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Ranger21: a synergistic deep learning optimizer
• Who Are We Missing? A Principled Approach to Characterizing the Underrepresented Population
• BERT-JAM: Boosting BERT-Enhanced Neural Machine Translation with Joint Attention
• LRTuner: A Learning Rate Tuner for Deep Neural Networks