Towards Understanding the Condensation of Neural Networks at Initial Training

Hanxu Zhou, Qixuan Zhou, Tao Luo +2

Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from…