A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
arXiv:1810.00122
Abstract
Despite its empirical success and recent theoretical progress, there generally lacks a quantitative analysis of the effect of batch normalization (BN) on the convergence and stability of gradient descent. In this paper, we provide such an analysis on the simple problem of ordinary least squares (OLS). Since precise dynamical properties of gradient descent (GD) is completely known for the OLS problem, it allows us to isolate and compare the additional effects of BN. More precisely, we show that unlike GD, gradient descent with BN (BNGD) converges for arbitrary learning rates for the weights, and the convergence remains linear under mild conditions. Moreover, we quantify two different sources of acceleration of BNGD over GD -- one due to over-parameterization which improves the effective condition number and another due having a large range of learning rates giving rise to fast descent. These phenomena set BNGD apart from GD and could account for much of its robustness properties. These findings are confirmed quantitatively by numerical experiments, which further show that many of the uncovered properties of BNGD in OLS are also observed qualitatively in more complex supervised learning problems.
References in corpus (6)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Gradient Descent Converges to Minimizers
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization