An Exponential Learning Rate Schedule for Deep Learning

Abstract

Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN. 1. Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i.e., learning rate increases by some $(1+\alpha )$ factor in every epoch for some $\alpha >0$. (Precise statement in the paper.) To the best of our knowledge this is the first time such a rate schedule has been successfully used, let alone for highly successful architectures. As expected, such training rapidly blows up network weights, but the net stays well-behaved due to normalization. 2. Mathematical explanation of the success of the above rate schedule: a rigorous proof that it is equivalent to the standard setting of BN + SGD + StandardRate Tuning + Weight Decay + Momentum. This equivalence holds for other normalization layers as well, Group Normalization, LayerNormalization, Instance Norm, etc. 3. A worked-out toy example illustrating the above linkage of hyper-parameters. Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Large Batch Training of Convolutional Networks
• L2 Regularization versus Batch and Weight Normalization
• Norm matters: efficient and accurate normalization schemes in deep networks
• WNGrad: Learn the Learning Rate in Gradient Descent
• SGD: General Analysis and Improved Rates
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

Cited by in corpus (23)

• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Facial Emotion Recognition: State of the Art Performance on FER2013
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Bad Global Minima Exist and SGD Can Reach Them
• On the training dynamics of deep networks with $L_2$ regularization
• Regular Polytope Networks
• Angle-based Search Space Shrinking for Neural Architecture Search
• How to decay your learning rate
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• On Learning Rates and Schrödinger Operators
• The Two Regimes of Deep Network Training
• An Unconstrained Layer-Peeled Perspective on Neural Collapse
• Disentangling Adaptive Gradient Methods from Learning Rates
• Formal Language Theory Meets Modern NLP
• Stochastic Optimization with Laggard Data Pipelines
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• Rethinking Experience Replay: a Bag of Tricks for Continual Learning
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Acceleration via Fractal Learning Rate Schedules
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• S3Net: 3D LiDAR Sparse Semantic Segmentation Network
• Adaptive Gradient Method with Resilience and Momentum