282 citations · 989 across the 11 of their papers we have counts for
Showing 2019Show all
3 papers · 1 filter
cs.LG2019★ 2 cited
Effects of Depth, Width, and Initialization: A Convergence Analysis of Layer-wise Training for Deep Linear Neural Networks
Yeonjong Shin
Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep neural networks is a challenging ta…
cs.LG2019
Trainability of ReLU networks and Data-dependent Initialization
Yeonjong Shin, George Em Karniadakis
In this paper, we study the trainability of rectified linear unit (ReLU) networks. A ReLU neuron is said to be dead if it only outputs a constant for any input. Two death states of…
stat.ML2019★ 282 cited
Dying ReLU and Initialization: Theory and Numerical Examples
Lu Lu, Yeonjong Shin, Yanhui Su +1
The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die.…