Dying ReLU and Initialization: Theory and Numerical Examples

Lu Lu, Yeonjong Shin, Yanhui Su +1

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die.…