Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

Abstract

The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. The purpose of this work is to propose a modern view and a general mathematical framework for loss landscapes and efficient optimization in over-parameterized machine learning models and systems of non-linear equations, a setting that includes over-parameterized deep neural networks. Our starting observation is that optimization problems corresponding to such systems are generally not convex, even locally. We argue that instead they satisfy PL${}^{\ast }$, a variant of the Polyak-Lojasiewicz condition on most (but not all) of the parameter space, which guarantees both the existence of solutions and efficient optimization by (stochastic) gradient descent (SGD/GD). The PL${}^{\ast }$ condition of these systems is closely related to the condition number of the tangent kernel associated to a non-linear system showing how a PL${}^{\ast }$-based non-linear theory parallels classical analyses of over-parameterized linear equations. We show that wide neural networks satisfy the PL${}^{\ast }$ condition, which explains the (S)GD convergence to a global minimum. Finally we propose a relaxation of the PL${}^{\ast }$ condition applicable to "almost" over-parameterized systems.

The discussion on transition to linearity in Version 1 has been moved to arXiv:2010.01092 (appeared in NeurIPS 2020)

References in corpus (7)

• Language Models are Few-Shot Learners
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Double descent in the condition number
• Path Length Bounds for Gradient Descent and Flow

Cited by in corpus (3)

• Universal Effectiveness of High-Depth Circuits in Variational Eigenproblems
• Learning with Gradient Descent and Weakly Convex Losses
• Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent