Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

Hyungu Lee, Taehyeong Kim, Hayoung Choi

Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying…