Effect of the initial configuration of weights on the training and function of artificial neural networks

Abstract

The function and performance of neural networks is largely determined by the evolution of their weights and biases in the process of training, starting from the initial configuration of these parameters to one of the local minima of the loss function. We perform the quantitative statistical characterization of the deviation of the weights of two-hidden-layer ReLU networks of various sizes trained via Stochastic Gradient Descent (SGD) from their initial random configuration. We compare the evolution of the distribution function of this deviation with the evolution of the loss during training. We observed that successful training via SGD leaves the network in the close neighborhood of the initial configuration of its weights. For each initial weight of a link we measured the distribution function of the deviation from this value after training and found how the moments of this distribution and its peak depend on the initial weight. We explored the evolution of these deviations during training and observed an abrupt increase within the overfitting region. This jump occurs simultaneously with a similarly abrupt increase recorded in the evolution of the loss function. Our results suggest that SGD's ability to efficiently find local minima is restricted to the vicinity of the random initial configuration of weights.

arXiv comment12 pages, 8 figures

References in corpus (17)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• On the Convergence Rate of Training Recurrent Neural Networks
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• The Early Phase of Neural Network Training
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• The HASYv2 dataset