Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
arXiv:2110.03141
Abstract
Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts SAM s efficiency at no cost to its generalization performance. ESAM includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. In the former, the sharpness measure is approximated by perturbing a stochastically chosen set of weights in each iteration; in the latter, the SAM loss is optimized using only a judiciously selected subset of data that is sensitive to the sharpness. We provide theoretical explanations as to why these strategies perform well. We also show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis base optimizers, while test accuracies are preserved or even improved.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Wide Residual Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Visualizing the Loss Landscape of Neural Nets
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Lookahead Optimizer: k steps forward, 1 step back
- Exploring Generalization in Deep Learning
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Adversarial Weight Perturbation Helps Robust Generalization
- Fantastic Generalization Measures and Where to Find Them
- Sharp Minima Can Generalize For Deep Nets
- Sharpness-Aware Minimization for Efficiently Improving Generalization
- When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
- On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
- VOLO: Vision Outlooker for Visual Recognition
- ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
- Label Noise SGD Provably Prefers Flat Global Minimizers
- Using Early-Learning Regularization to Classify Real-World Noisy Data