A Study of Gradient Variance in Deep Learning

Abstract

The impact of gradient noise on training deep models is widely acknowledged but not well understood. In this context, we study the distribution of gradients during training. We introduce a method, Gradient Clustering, to minimize the variance of average mini-batch gradient with stratified sampling. We prove that the variance of average mini-batch gradient is minimized if the elements are sampled from a weighted clustering in the gradient space. We measure the gradient variance on common deep learning benchmarks and observe that, contrary to common assumptions, gradient variance increases during training, and smaller learning rates coincide with higher variance. In addition, we introduce normalized gradient variance as a statistic that better correlates with the speed of convergence compared to gradient variance.

References in corpus (17)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Revisiting Small Batch Training for Deep Neural Networks
• New insights and perspectives on the natural gradient method
• The generalization error of random features regression: Precise asymptotics and double descent curve
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Measuring the Effects of Data Parallelism on Neural Network Training
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling
• Are All Training Examples Created Equal? An Empirical Study
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• Accelerating Deep Learning by Focusing on the Biggest Losers
• Efficient Per-Example Gradient Computations
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Pathological spectra of the Fisher information metric and its variants in deep neural networks

Cited by in corpus (5)

• An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
• A Tale Of Two Long Tails
• Training Efficiency and Robustness in Deep Learning