Towards Understanding the Importance of Noise in Training Neural Networks

Abstract

Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks. The theory behind, however, is still largely unknown. This paper studies this fundamental problem through training a simple two-layer convolutional neural network model. Although training such a network requires solving a nonconvex optimization problem with a spurious local optimum and a global optimum, we prove that perturbed gradient descent and perturbed mini-batch stochastic gradient algorithms in conjunction with noise annealing is guaranteed to converge to a global optimum in polynomial time with arbitrary initialization. This implies that the noise enables the algorithm to efficiently escape from the spurious local optimum. Numerical experiments are provided to support our theory.

arXiv commentInternational Conference on Machine Learning (ICML), 2019

Cited by in corpus (7)

• Speech Denoising Without Clean Training Data: A Noise2Noise Approach
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting
• A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization
• Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics