ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks

Abstract

Recently, learning algorithms motivated from sharpness of loss surface as an effective measure of generalization gap have shown state-of-the-art performances. Nevertheless, sharpness defined in a rigid region with a fixed radius, has a drawback in sensitivity to parameter re-scaling which leaves the loss unaffected, leading to weakening of the connection between sharpness and generalization gap. In this paper, we introduce the concept of adaptive sharpness which is scale-invariant and propose the corresponding generalization bound. We suggest a novel learning method, adaptive sharpness-aware minimization (ASAM), utilizing the proposed generalization bound. Experimental results in various benchmark datasets show that ASAM contributes to significant improvement of model generalization performance.

arXiv comment13 pages, 4 figures, To be published in ICML 2021

References in corpus (11)

• Improved Regularization of Convolutional Neural Networks with Cutout
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Exploring Generalization in Deep Learning
• ShakeDrop Regularization for Deep Residual Learning
• Sharp Minima Can Generalize For Deep Nets
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Training Recurrent Neural Networks by Diffusion
• Positively Scale-Invariant Flatness of ReLU Neural Networks
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption

Cited by in corpus (2)

• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks