Sample Variance Decay in Randomly Initialized ReLU Networks

Kyle Luther, H. Sebastian Seung

Before training a neural net, a classic rule of thumb is to randomly initialize the weights so the variance of activations is preserved across layers. This is traditionally interpr…