Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron

Jun-Kun Wang, Jacob Abernethy

Over-parametrization has become a popular technique in deep learning. It is observed that by over-parametrization, a larger neural network needs a fewer training iterations than a…