Spurious Local Minima are Common in Two-Layer ReLU Neural Networks

Itay Safran, Ohad Shamir

We consider the optimization problem associated with training simple ReLU neural networks of the form $\mathbf{x}\mapsto {\sum }_{i=1}^k\max \{0,{\mathbf{w}}_i^{\top }\mathbf{x}\}$ with…