450 citations · 557 across the 27 of their papers we have counts for
Showing 2021Show all
3 papers · 1 filter
cs.LG2021
Activation function design for deep networks: linearity and effective initialisation
Michael Murray, Vinayak Abrol, Jared Tanner
The activation function deployed in a deep neural network has great influence on the performance of the network at initialisation, which in turn has implications for training. In t…
cs.IT2021
Mutual Information of Neural Network Initialisations: Mean Field Approximations
Jared Tanner, Giuseppe Ughi
The ability to train randomly initialised deep neural networks is known to depend strongly on the variance of the weight matrices and biases as well as the choice of nonlinear acti…
cs.LG2021
Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
Ilan Price, Jared Tanner
That neural networks may be pruned to high sparsities and retain high accuracy is well established. Recent research efforts focus on pruning immediately after initialization so as…