On weight initialization in deep neural networks

Abstract

A proper initialization of the weights in a neural network is critical to its convergence. Current insights into weight initialization come primarily from linear activation functions. In this paper, I develop a theory for weight initializations with non-linear activations. First, I derive a general weight initialization strategy for any neural network using activation functions differentiable at 0. Next, I derive the weight initialization strategy for the Rectified Linear Unit (RELU), and provide theoretical insights into why the Xavier initialization is a poor choice with RELU activations. My analysis provides a clear demonstration of the role of non-linearities in determining the proper weight initializations.

arXiv comment9 pages, 4 figures

References in corpus (2)

• Sequence to Sequence Learning with Neural Networks
• Convolutional Neural Networks for Sentence Classification

Cited by in corpus (28)

• Deep Learning in fNIRS: A review
• Review: Deep Learning in Electron Microscopy
• Kolmogorov-Arnold Recurrent Network for Short Term Load Forecasting Across Diverse Consumers
• On the Use of Neural Networks for Full Waveform Inversion
• Underwater Acoustic Communication Channel Modeling using Reservoir Computing
• Exploring Benefits of Transfer Learning in Neural Machine Translation
• A Survey on Activation Functions and their relation with Xavier and He Normal Initialization
• Leveraging Hypernetworks and Learnable Kernels for Consumer Energy Forecasting Across Diverse Consumer Types
• AdjointNet: Constraining machine learning models with physics-based codes
• One-Bit Compressive Sensing: Can We Go Deep and Blind?
• Filter Grafting for Deep Neural Networks
• Transfer Learning Enhanced Full Waveform Inversion
• Gradient target propagation
• Clusterability in Neural Networks
• FlexPINN: Modeling Fluid Dynamics and Mass Transfer in 3D Micromixer Geometries Using a Flexible Physics-Informed Neural Network
• Detecting Dead Weights and Units in Neural Networks
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• Grassmannian Constellation Design for Noncoherent MIMO Systems Using Autoencoders
• CompNet: Neural networks growing via the compact network morphism
• Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception
• LiDAR ICPS-net: Indoor Camera Positioning based-on Generative Adversarial Network for RGB to Point-Cloud Translation
• Detection and classification of vocal productions in large scale audio recordings
• Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks
• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis
• Filter Grafting for Deep Neural Networks: Reason, Method, and Cultivation
• A Survey on Green Deep Learning
• Spatial light interference microscopy (SLIM): principle and applications to biomedicine
• Online-Learning Deep Neuro-Adaptive Dynamic Inversion Controller for Model Free Control