Inherent Noise in Gradient Based Methods

Abstract

Previous work has examined the ability of larger capacity neural networks to generalize better than smaller ones, even without explicit regularizers, by analyzing gradient based algorithms such as GD and SGD. The presence of noise and its effect on robustness to parameter perturbations has been linked to generalization. We examine a property of GD and SGD, namely that instead of iterating through all scalar weights in the network and updating them one by one, GD (and SGD) updates all the parameters at the same time. As a result, each parameter $w^i$ calculates its partial derivative at the stale parameter ${\mathbf{w}}_{\mathbf{t}}$, but then suffers loss $\hat{L}({\mathbf{w}}_{\mathbf{t}+1})$. We show that this causes noise to be introduced into the optimization. We find that this noise penalizes models that are sensitive to perturbations in the weights. We find that penalties are most pronounced for batches that are currently being used to update, and are higher for larger models.

References in corpus (11)

• Weight Uncertainty in Neural Networks
• Understanding deep learning requires rethinking generalization
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Dropout Training as Adaptive Regularization
• Three Factors Influencing Minima in SGD
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Sharp Minima Can Generalize For Deep Nets
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study