Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"

Abstract

We revisit and extend the experiments of Goodfellow et al. (2014), who showed that - for then state-of-the-art networks - "the objective function has a simple, approximately convex shape" along the linear path between initialization and the trained weights. We do not find this to be the case for modern networks on CIFAR-10 and ImageNet. Instead, although loss is roughly monotonically non-increasing along this path, it remains high until close to the optimum. In addition, training quickly becomes linearly separated from the optimum by loss barriers. We conclude that, although Goodfellow et al.'s findings describe the "relatively easy to optimize" MNIST setting, behavior is qualitatively different in modern settings.

Workshop on Deep Learning and Information Geometry (NeurIPS 2020)

References in corpus (1)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Cited by in corpus (5)

• Connecting Low-Loss Subspace for Personalized Federated Learning
• Learning Neural Network Subspaces
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• On the topology of solutions to random continuous constraint satisfaction problems
• What can linear interpolation of neural network loss landscapes tell us?