Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations

Tian Ding, Dawei Li, Ruoyu Sun

Does over-parameterization eliminate sub-optimal local minima for neural networks? An affirmative answer was given by a classical result in [59] for 1-hidden-layer wide neural netw…