Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
arXiv:2006.08419
Abstract
In this work, we comprehensively reveal the learning dynamics of neural network with normalization, weight decay (WD), and SGD (with momentum), named as Spherical Motion Dynamics (SMD). Most related works study SMD by focusing on "effective learning rate" in "equilibrium" condition, where weight norm remains unchanged. However, their discussions on why equilibrium condition can be reached in SMD is either absent or less convincing. Our work investigates SMD by directly exploring the cause of equilibrium condition. Specifically, 1) we introduce the assumptions that can lead to equilibrium condition in SMD, and prove that weight norm can converge at linear rate with given assumptions; 2) we propose "angular update" as a substitute for effective learning rate to measure the evolving of neural network in SMD, and prove angular update can also converge to its theoretical value at linear rate; 3) we verify our assumptions and theoretical results on various computer vision tasks including ImageNet and MSCOCO with standard settings. Experiment results show our theoretical findings agree well with empirical observations.
arXiv commentTheoretical analysis on joint effect of normalization and weight decay
References in corpus (15)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- How Does Batch Normalization Help Optimization?
- L2 Regularization versus Batch and Weight Normalization
- Norm matters: efficient and accurate normalization schemes in deep networks
- Escaping Saddles with Stochastic Gradients
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
- Towards Understanding Regularization in Batch Normalization
- Online Normalization for Training Neural Networks
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
- Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
- Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs