Pure and Spurious Critical Points: a Geometric Study of Linear Networks

Abstract

The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights. We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, which arise from the parameterization. We apply this perspective to revisit and extend the literature on the loss function of linear neural networks. For this type of network, the functional space is either the set of all linear maps from input to output space, or a determinantal variety, i.e., a set of linear maps with bounded rank. We use geometric properties of determinantal varieties to derive new results on the landscape of linear networks with different loss functions and different parameterizations. Our analysis clearly illustrates that the absence of "bad" local minima in the loss landscape of linear networks is due to two distinct phenomena that apply in different settings: it is true for arbitrary smooth convex losses in the case of architectures that can express all linear maps ("filling architectures") but it holds only for the quadratic loss when the functional space is a determinantal variety ("non-filling architectures"). Without any assumption on the architecture, smooth convex losses may lead to landscapes with many bad minima.

References in corpus (13)

• The Loss Surfaces of Multilayer Networks
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Depth Creates No Bad Local Minima
• Implicit Regularization in Deep Matrix Factorization
• Global optimality conditions for deep neural networks
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• On the Expressive Power of Deep Polynomial Neural Networks
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties
• The Euclidean distance degree of an algebraic variety
• Depth creates no more spurious local minima
• Deep linear neural networks with arbitrary loss: All local minima are global
• The average number of critical rank-one approximations to a tensor

Cited by in corpus (2)

• The loss landscape of deep linear neural networks: a second-order analysis
• Training Linear Neural Networks: Non-Local Convergence and Complexity Results