Sharpness-Aware Minimization for Efficiently Improving Generalization

Abstract

In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-10, CIFAR-100, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. We open source our code at \url{https://github.com/google-research/sam}.

References in corpus (8)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Improved Regularization of Convolutional Neural Networks with Cutout
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Improving Generalization Performance by Switching from Adam to SGD
• Shake-Shake regularization
• Unsupervised Label Noise Modeling and Loss Correction
• Fantastic Generalization Measures and Where to Find Them

Cited by in corpus (22)

• Transformers in Vision: A Survey
• Combining EfficientNet and Vision Transformers for Video Deepfake Detection
• User-friendly introduction to PAC-Bayes bounds
• ResNet strikes back: An improved training procedure in timm
• Efficient Adaptive Ensembling for Image Classification
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning
• Learning Neural Network Subspaces
• Calibrated Adaptive Teacher for Domain Adaptive Intelligent Fault Diagnosis
• SIRe-Networks: Convolutional Neural Networks Architectural Extension for Information Preservation via Skip/Residual Connections and Interlaced Auto-Encoders
• Boosting Light-Weight Depth Estimation Via Knowledge Distillation
• Early Detection of Alzheimer's Disease using Bottleneck Transformers
• Adversarial Parameter Defense by Multi-Step Risk Minimization
• COVID Detection in Chest CTs: Improving the Baseline on COV19-CT-DB
• Graceful Degradation and Related Fields
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performance
• The Efficiency Misnomer
• Exact Stochastic Second Order Deep Learning
• Minimum sharpness: Scale-invariant parameter-robustness of neural networks