Ill-Posedness and Optimization Geometry for Nonlinear Neural Network Training

Abstract

In this work we analyze the role nonlinear activation functions play at stationary points of dense neural network training problems. We consider a generic least squares loss function training formulation. We show that the nonlinear activation functions used in the network construction play a critical role in classifying stationary points of the loss landscape. We show that for shallow dense networks, the nonlinear activation function determines the Hessian nullspace in the vicinity of global minima (if they exist), and therefore determines the ill-posedness of the training problem. Furthermore, for shallow nonlinear networks we show that the zeros of the activation function and its derivatives can lead to spurious local minima, and discuss conditions for strict saddle points. We extend these results to deep dense neural networks, showing that the last activation function plays an important role in classifying stationary points, due to how it shows up in the gradient from the chain rule.

References in corpus (4)

• How to Escape Saddle Points Efficiently
• Efficient approaches for escaping higher order saddle points in non-convex optimization
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training

Cited by in corpus (1)

• Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization