Improving Generalization Performance by Switching from Adam to SGD

Abstract

Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection of Adam steps on the gradient subspace. By design, the monitoring process for this condition adds very little overhead and does not increase the number of hyperparameters in the optimizer. We report experiments on several standard benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and CIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling with recurrent networks on the PTB and WT2 data sets. The results show that our strategy is capable of closing the generalization gap between SGD and Adam on a majority of the tasks.

References in corpus (3)

• Neural Machine Translation by Jointly Learning to Align and Translate
• On the Convergence of Adam and Beyond
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

Cited by in corpus (92)

• On Empirical Comparisons of Optimizers for Deep Learning
• Optimization for deep learning: theory and algorithms
• Learning to Demodulate from Few Pilots via Offline and Online Meta-Learning
• AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Optimize TSK Fuzzy Systems for Classification Problems: Mini-Batch Gradient Descent with Uniform Regularization and Batch Normalization
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• Large Kernel Distillation Network for Efficient Single Image Super-Resolution
• Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• A Survey of Optimization Methods from a Machine Learning Perspective
• The Heavy-Tail Phenomenon in SGD
• Toward Interpretable Music Tagging with Self-Attention
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Physics-constrained Deep Learning for Robust Inverse ECG Modeling
• An Adaptive and Momental Bound Method for Stochastic Learning
• AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
• The Implicit and Explicit Regularization Effects of Dropout
• Exploring Data Augmentation for Multi-Modality 3D Object Detection
• GADAM: Genetic-Evolutionary ADAM for Deep Neural Network Optimization
• Black-Box Optimization with Local Generative Surrogates
• On the adequacy of untuned warmup for adaptive optimization
• Mixing ADAM and SGD: a Combined Optimization Method
• Optimizer Benchmarking Needs to Account for Hyperparameter Tuning
• Music theme recognition using CNN and self-attention
• Learning to play the Chess Variant Crazyhouse above World Champion Level with Deep Neural Networks and Human Data
• Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks
• On the Generalization of Stochastic Gradient Descent with Momentum
• When Does Preconditioning Help or Hurt Generalization?
• Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models
• An Adaptive Gradient Method with Energy and Momentum
• Disentangling Adaptive Gradient Methods from Learning Rates
• SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
• Optimal Adaptive and Accelerated Stochastic Gradient Descent
• Characterizing Structural Regularities of Labeled Data in Overparameterized Models
• To Each Optimizer a Norm, To Each Norm its Generalization
• Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
• Versatile and Robust Transient Stability Assessment via Instance Transfer Learning
• Human Motion Anticipation with Symbolic Label
• Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation
• Deep Shape-from-Template: Wide-Baseline, Dense and Fast Registration and Deformable Reconstruction from a Single Image
• Conformal Symplectic Optimization for Stable Reinforcement Learning
• AdaSGD: Bridging the gap between SGD and Adam
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• Light-Weighted CNN for Text Classification
• Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization
• Projection-Free Adaptive Gradients for Large-Scale Optimization
• Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks
• Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification
• Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
• Multi Task Deep Morphological Analyzer: Context Aware Joint Morphological Tagging and Lemma Prediction
• Orthogonal Over-Parameterized Training
• Information-Theoretic Local Minima Characterization and Regularization
• Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum
• Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties
• Stage-based Hyper-parameter Optimization for Deep Learning
• Flatness is a False Friend
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
• Loop Closure Detection with RGB-D Feature Pyramid Siamese Networks
• Smoothness Analysis of Adversarial Training
• Stochastic Gradient Methods with Block Diagonal Matrix Adaptation
• Visual Question Answering as a Multi-Task Problem
• VTLayout: Fusion of Visual and Text Features for Document Layout Analysis
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• Structured Dropout Variational Inference for Bayesian Neural Networks
• DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization
• BERT-based Chinese Text Classification for Emergency Domain with a Novel Loss Function
• "You eat with your eyes first": Optimizing Yelp Image Advertising
• Statistical Measures For Defining Curriculum Scoring Function
• A Dynamic Sampling Adaptive-SGD Method for Machine Learning
• CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity
• Channel Assignment in Uplink Wireless Communication using Machine Learning Approach
• Eliminating Catastrophic Interference with Biased Competition
• Dosimetric impact of physician style variations in contouring CTV for post-operative prostate cancer: A deep learning-based simulation study
• ACMo: Angle-Calibrated Moment Methods for Stochastic Optimization
• Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
• Flexible numerical optimization with ensmallen
• TAG: Task-based Accumulated Gradients for Lifelong learning
• AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy
• Toward Communication Efficient Adaptive Gradient Method
• Local Convergence of Adaptive Gradient Descent Optimizers
• Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods
• Exploring the efficacy of neural networks for trajectory compression and the inverse problem
• The Channel Attention based Context Encoder Network for Inner Limiting Membrane Detection
• Variable-sized input, character-level recurrent neural networks in lead generation: predicting close rates from raw user inputs
• Optimizing Convergence for Iterative Learning of ARIMA for Stationary Time Series
• An Adaptive Remote Stochastic Gradient Method for Training Neural Networks