Sharpness-Aware Minimization for Efficiently Improving Generalization

Abstract

In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-10, CIFAR-100, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. We open source our code at \url{https://github.com/google-research/sam}.

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Deep Residual Learning for Image Recognition
• Improved Regularization of Convolutional Neural Networks with Cutout
• Wide Residual Networks
• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• AutoAugment: Learning Augmentation Policies from Data
• Visualizing the Loss Landscape of Neural Nets
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Improving Generalization Performance by Switching from Adam to SGD
• Shake-Shake regularization
• Deep Networks with Stochastic Depth
• Averaging Weights Leads to Wider Optima and Better Generalization
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Adversarial Weight Perturbation Helps Robust Generalization
• Unsupervised Label Noise Modeling and Loss Correction
• Fantastic Generalization Measures and Where to Find Them
• FMix: Enhancing Mixed Sample Data Augmentation
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Domain Adaptive Transfer Learning with Specialist Models
• Deep Pyramidal Residual Networks
• Training Recurrent Neural Networks by Diffusion
• Multi-branch and Multi-scale Attention Learning for Fine-Grained Visual Categorization
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption

Cited by in corpus (24)

• Transformers in Vision: A Survey
• Combining EfficientNet and Vision Transformers for Video Deepfake Detection
• User-friendly introduction to PAC-Bayes bounds
• ResNet strikes back: An improved training procedure in timm
• Efficient Adaptive Ensembling for Image Classification
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning
• Learning Neural Network Subspaces
• Calibrated Adaptive Teacher for Domain Adaptive Intelligent Fault Diagnosis
• Drive-specific adaptation in disordered mechanical networks of bistable springs
• SIRe-Networks: Convolutional Neural Networks Architectural Extension for Information Preservation via Skip/Residual Connections and Interlaced Auto-Encoders
• Boosting Light-Weight Depth Estimation Via Knowledge Distillation
• Early Detection of Alzheimer's Disease using Bottleneck Transformers
• Adversarial Parameter Defense by Multi-Step Risk Minimization
• COVID Detection in Chest CTs: Improving the Baseline on COV19-CT-DB
• Graceful Degradation and Related Fields
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• The Renyi Gaussian Process: Towards Improved Generalization
• HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performance
• Exact Stochastic Second Order Deep Learning
• Minimum sharpness: Scale-invariant parameter-robustness of neural networks
• The Efficiency Misnomer