Improved generalization by noise enhancement

Abstract

Recent studies have demonstrated that noise in stochastic gradient descent (SGD) is closely related to generalization: A larger SGD noise, if not too large, results in better generalization. Since the covariance of the SGD noise is proportional to ${\eta }^2/B$, where $\eta$ is the learning rate and $B$ is the minibatch size of SGD, the SGD noise has so far been controlled by changing $\eta$ and/or $B$. However, too large $\eta$ results in instability in the training dynamics and a small $B$ prevents scalable parallel computation. It is thus desirable to develop a method of controlling the SGD noise without changing $\eta$ and $B$. In this paper, we propose a method that achieves this goal using ``noise enhancement'', which is easily implemented in practice. We expound the underlying theoretical idea and demonstrate that the noise enhancement actually improves generalization for real datasets. It turns out that large-batch training with the noise enhancement even shows better generalization compared with small-batch training.

arXiv comment9 pages

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• One weird trick for parallelizing convolutional neural networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• On Exact Computation with an Infinitely Wide Neural Net
• Three Factors Influencing Minima in SGD
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Generalization in Deep Networks: The Role of Distance from Initialization
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Augment your batch: better training with larger batches
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Non-Gaussianity of Stochastic Gradient Noise

Cited by in corpus (1)

• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent