Initializing ReLU networks in an expressive subspace of weights

Abstract

Using a mean-field theory of signal propagation, we analyze the evolution of correlations between two signals propagating forward through a deep ReLU network with correlated weights. Signals become highly correlated in deep ReLU networks with uncorrelated weights. We show that ReLU networks with anti-correlated weights can avoid this fate and have a chaotic phase where the signal correlations saturate below unity. Consistent with this analysis, we find that networks initialized with anti-correlated weights can train faster (in a teacher-student setting) by taking advantage of the increased expressivity in the chaotic phase. Combining this with a previously proposed strategy of using an asymmetric initialization to reduce dead node probability, we propose an initialization scheme that allows faster training and learning than the best-known initializations.

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ADADELTA: An Adaptive Learning Rate Method
• Gaussian Error Linear Units (GELUs)
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Training Very Deep Networks
• Deep Neural Networks as Gaussian Processes
• Layer Normalization
• Exponential expressivity in deep neural networks through transient chaos
• On the Spectral Bias of Neural Networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• On the Impact of the Activation Function on Deep Neural Networks Training
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Random deep neural networks are biased towards simple functions
• Gradient Descent Quantizes ReLU Network Features
• Exploring the Function Space of Deep-Learning Machines