Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysis

Alexander Munteanu, Simon Omlor, Zhao Song +1

A common method in training neural networks is to initialize all the weights to be independent Gaussian vectors. We observe that by instead initializing the weights into independen…