Dying ReLU and Initialization: Theory and Numerical Examples

Abstract

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in probability as the depth goes to infinite. Several methods have been proposed to alleviate the dying ReLU. Perhaps, one of the simplest treatments is to modify the initialization procedure. One common way of initializing weights and biases uses symmetric probability distributions, which suffers from the dying ReLU. We thus propose a new initialization procedure, namely, a randomized asymmetric initialization. We prove that the new initialization can effectively prevent the dying ReLU. All parameters required for the new initialization are theoretically designed. Numerical examples are provided to demonstrate the effectiveness of the new initialization procedure.

Cited by in corpus (23)

• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Unsupervised Deep Learning for Massive MIMO Hybrid Beamforming
• ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognition
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• Combining data assimilation and machine learning to estimate parameters of a convective-scale model
• CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality
• All-optical nonlinear activation function based on stimulated Brillouin scattering
• Modelling the galaxy-halo connection with semi-recurrent neural networks
• A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Quantum DeepONet: Neural operators accelerated by quantum computing
• Phishing Detection in the Gen-AI Era: Quantized LLMs vs Classical Models
• Polynomial Ridge Flowfield Estimation
• Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks
• A discrete physics-informed training for projection-based reduced order models with neural networks
• REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs
• EffCNet: An Efficient CondenseNet for Image Classification on NXP BlueBox
• Bayesian optimization approach for tracking the location and orientation of a moving target using far-field data
• A deep learning approach to the texture optimization problem for friction control in lubricated contacts
• Leaky ReLUs That Differ in Forward and Backward Pass Facilitate Activation Maximization in Deep Neural Networks