Gaussian Pre-Activations in Neural Networks: Myth or Reality?

Pierre Wolinski, Julyan Arbel

The study of feature propagation at initialization in neural networks lies at the root of numerous initialization designs. An assumption very commonly made in the field states that…