Fractional moment-preserving initialization schemes for training deep neural networks

Abstract

A traditional approach to initialization in deep neural networks (DNNs) is to sample the network weights randomly for preserving the variance of pre-activations. On the other hand, several studies show that during the training process, the distribution of stochastic gradients can be heavy-tailed especially for small batch sizes. In this case, weights and therefore pre-activations can be modeled with a heavy-tailed distribution that has an infinite variance but has a finite (non-integer) fractional moment of order $s$ with $s<2$. Motivated by this fact, we develop initialization schemes for fully connected feed-forward networks that can provably preserve any given moment of order $s\in (0,2]$ over the layers for a class of activations including ReLU, Leaky ReLU, Randomized Leaky ReLU, and linear activations. These generalized schemes recover traditional initialization schemes in the limit $s\to 2$ and serve as part of a principled theory for initialization. For all these schemes, we show that the network output admits a finite almost sure limit as the number of layers grows, and the limit is heavy-tailed in some settings. This sheds further light into the origins of heavy tail during signal propagation in DNNs. We prove that the logarithm of the norm of the network outputs, if properly scaled, will converge to a Gaussian distribution with an explicit mean and variance we can compute depending on the activation used, the value of s chosen and the network width. We also prove that our initialization scheme avoids small network output values more frequently compared to traditional approaches. Furthermore, the proposed initialization strategy does not have an extra cost during the training procedure. We show through numerical experiments that our initialization can improve the training and test performance.

References in corpus (15)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• An Empirical Evaluation of Deep Learning on Highway Driving
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Analyzing noise in autoencoders and deep networks
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Traditional and Heavy-Tailed Self Regularization in Neural Network Models
• The Heavy-Tail Phenomenon in SGD
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Non-Gaussianity of Stochastic Gradient Noise
• Critical initialisation for deep signal propagation in noisy rectifier neural networks
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Sample Variance Decay in Randomly Initialized ReLU Networks

Cited by in corpus (2)

• Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections