How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

Devansh Arpit, Victor Campos, Yoshua Bengio

Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied…