SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization

Abstract

In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.

References in corpus (5)

• The Loss Surfaces of Multilayer Networks
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Improving Generalization Performance by Switching from Adam to SGD
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• A Scale Invariant Flatness Measure for Deep Network Minima