A Note on Connectivity of Sublevel Sets in Deep Learning

Abstract

It is shown that for deep neural networks, a single wide layer of width $N+1$ ($N$ being the number of training samples) suffices to prove the connectivity of sublevel sets of the training loss function. In the two-layer setting, the same property may not hold even if one has just one neuron less (i.e. width $N$ can lead to disconnected sublevel sets).

References in corpus (5)

• The Loss Surfaces of Multilayer Networks
• Optimization Landscape and Expressivity of Deep CNNs
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Topology and Geometry of Half-Rectified Network Optimization
• Small nonlinearities in activation functions create bad local minima in neural networks

Cited by in corpus (2)

• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• When Are Solutions Connected in Deep Networks?