Qualitatively characterizing neural network optimization problems

Abstract

Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and other obstacles motivating a variety of schemes to improve optimization, such as unsupervised pretraining. However, modern neural networks are able to achieve negligible training error on complex tasks, using only direct training with stochastic gradient descent. We introduce a simple analysis technique to look for evidence that such networks are overcoming local optima. We find that, in fact, on a straight path from initialization to solution, a variety of state of the art neural networks never encounter any significant obstacles.

References in corpus (3)

• Explaining and Harnessing Adversarial Examples
• Recurrent Neural Network Regularization
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Cited by in corpus (6)

• Exploring loss function topology with cyclical learning rates
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Large Scale Structure of Neural Network Loss Landscapes
• Multiscale Hierarchical Convolutional Networks
• Spontaneous Symmetry Breaking in Neural Networks
• Are Saddles Good Enough for Deep Learning?