Mutual Information of Neural Network Initialisations: Mean Field Approximations

Jared Tanner, Giuseppe Ughi

The ability to train randomly initialised deep neural networks is known to depend strongly on the variance of the weight matrices and biases as well as the choice of nonlinear acti…