Critical initialisation in continuous approximations of binary neural networks

George Stamatescu, Federica Gerace, Carlo Lucibello +2

The training of stochastic neural network models with binary ($\pm 1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a no…