SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization

Abstract

In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.

References in corpus (20)

• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• The Loss Surfaces of Multilayer Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Improving Generalization Performance by Switching from Adam to SGD
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Sharp Minima Can Generalize For Deep Nets
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Identifying Generalization Properties in Neural Networks
• SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning
• A Scale Invariant Flatness Measure for Deep Network Minima
• Fast Rates for Empirical Risk Minimization of Strict Saddle Problems
• The Impact of Local Geometry and Batch Size on Stochastic Gradient Descent for Nonconvex Problems