The Goldilocks zone: Towards better understanding of neural network loss landscapes

Stanislav Fort, Adam Scherlis

We explore the loss landscape of fully-connected and convolutional neural networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating the Hessian, $H$, of the l…