Analytic expressions for the output evolution of a deep neural network

Abstract

We present a novel methodology based on a Taylor expansion of the network output for obtaining analytical expressions for the expected value of the network weights and output under stochastic training. Using these analytical expressions the effects of the hyperparameters and the noise variance of the optimization algorithm on the performance of the deep neural network are studied. In the early phases of training with a small noise coefficient, the output is equivalent to a linear model. In this case the network can generalize better due to the noise preventing the output from fully converging on the train data, however the noise does not result in any explicit regularization. In the later training stages, when higher order approximations are required, the impact of the noise becomes more significant, i.e. in a model which is non-linear in the weights noise can regularize the output function resulting in better generalization as witnessed by its influence on the weight Hessian, a commonly used metric for generalization capabilities.

References in corpus (13)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Robust Large Margin Deep Neural Networks
• On Lazy Training in Differentiable Programming
• Three Factors Influencing Minima in SGD
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Kernel and Rich Regimes in Overparametrized Models
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
• Quantitative Weak Convergence for Discrete Stochastic Processes