Averaging Weights Leads to Wider Optima and Better Generalization

Abstract

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

arXiv commentAppears at the Conference on Uncertainty in Artificial Intelligence (UAI), 2018

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Wide Residual Networks
• The Loss Surfaces of Multilayer Networks
• Shake-Shake regularization
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Qualitatively characterizing neural network optimization problems
• Essentially No Barriers in Neural Network Energy Landscape
• Sharp Minima Can Generalize For Deep Nets
• Deep Pyramidal Residual Networks
• Exploring loss function topology with cyclical learning rates
• Variance Networks: When Expectation Does Not Meet Your Expectations

Cited by in corpus (66)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Lookahead Optimizer: k steps forward, 1 step back
• Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Smooth Adversarial Training
• DeepCMB: Lensing Reconstruction of the Cosmic Microwave Background with Deep Neural Networks
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• A neural network-based framework for financial model calibration
• Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• ResNet strikes back: An improved training procedure in timm
• VarifocalNet: An IoU-aware Dense Object Detector
• Understanding Generalization through Visualizations
• On the Validity of Bayesian Neural Networks for Uncertainty Estimation
• A community-powered search of machine learning strategy space to find NMR property prediction models
• Consensus Control for Decentralized Deep Learning
• Probabilistic Deep Learning to Quantify Uncertainty in Air Quality Forecasting
• Action and Perception as Divergence Minimization
• ScheduleNet: Learn to solve multi-agent scheduling problems with reinforcement learning
• Visualizing and Understanding the Effectiveness of BERT
• On the Reproducibility of Neural Network Predictions
• Learning Neural Network Subspaces
• Geometric Transformers for Protein Interface Contact Prediction
• LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classification
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• LaProp: Separating Momentum and Adaptivity in Adam
• POEM: Polarization of Embeddings for Domain-Invariant Representations
• SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation
• Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
• Bayesian Graph Neural Networks for Molecular Property Prediction
• Active and transfer learning with partially Bayesian neural networks for materials and chemicals
• Non-local NetVLAD Encoding for Video Classification
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections
• A strong baseline for image and video quality assessment
• A Hybrid Neural Network Model for Commonsense Reasoning
• PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Table Image Recognition to Latex
• Measuring and Harnessing Transference in Multi-Task Learning
• Using Mode Connectivity for Loss Landscape Analysis
• Nondeterminism and Instability in Neural Network Optimization
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
• FlexLip: A Controllable Text-to-Lip System
• The 2ST-UNet for Pneumothorax Segmentation in Chest X-Rays using ResNet34 as a Backbone for U-Net
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• Information-Theoretic Local Minima Characterization and Regularization
• Server Averaging for Federated Learning
• Implicit Neural Representations for Chemical Reaction Paths
• Why flatness does and does not correlate with generalization for deep neural networks
• Collaborative Deep Learning Across Multiple Data Centers
• Conversational Answer Generation and Factuality for Reading Comprehension Question-Answering
• Uncertainties of Satellite-based Essential Climate Variables from Deep Learning
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• ROI Regularization for Semi-supervised and Supervised Learning
• Gradient-based Data Augmentation for Semi-Supervised Learning
• End-to-End Training of CNN Ensembles for Person Re-Identification
• Data-informed Emulators for Multi-Physics Simulations
• Adversarial Training with Stochastic Weight Average
• Improving Adversarial Robustness for Free with Snapshot Ensemble
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training
• Offer Personalization using Temporal Convolution Network and Optimization
• Renofeation: A Simple Transfer Learning Method for Improved Adversarial Robustness