The Non-Linearity Perturbation Threshold: Width Scaling and Landscape Bifurcations in Deep Learning

Michael Alexander

We study how the optimization landscape of a neural network deforms as a non-linear activation is introduced through a smooth homotopy. Working first in an abstract local setting -…