When Are Solutions Connected in Deep Networks?

Abstract

The question of how and why the phenomenon of mode connectivity occurs in training deep neural networks has gained remarkable attention in the research community. From a theoretical perspective, two possible explanations have been proposed: (i) the loss function has connected sublevel sets, and (ii) the solutions found by stochastic gradient descent are dropout stable. While these explanations provide insights into the phenomenon, their assumptions are not always satisfied in practice. In particular, the first approach requires the network to have one layer with order of $N$ neurons ($N$ being the number of training samples), while the second one requires the loss to be almost invariant after removing half of the neurons at each layer (up to some rescaling of the remaining ones). In this work, we improve both conditions by exploiting the quality of the features at every intermediate layer together with a milder over-parameterization condition. More specifically, we show that: (i) under generic assumptions on the features of intermediate layers, it suffices that the last two hidden layers have order of $\sqrt{N}$ neurons, and (ii) if subsets of features at each layer are linearly separable, then no over-parameterization is needed to show the connectivity. Our experiments confirm that the proposed condition ensures the connectivity of solutions found by stochastic gradient descent, even in settings where the previous requirements do not hold.

arXiv commentAccepted at NeurIPS 2021

References in corpus (25)

• The Loss Surfaces of Multilayer Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Essentially No Barriers in Neural Network Energy Landscape
• The loss surface of deep and wide neural networks
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Depth Creates No Bad Local Minima
• Identity Matters in Deep Learning
• Diverse Neural Network Learns True Target Functions
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• Exponentially vanishing sub-optimal local minima in multilayer neural networks
• Optimization Landscape and Expressivity of Deep CNNs
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• Understanding the Loss Surface of Neural Networks for Binary Classification
• Adding One Neuron Can Eliminate All Bad Local Minima
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• Topology and Geometry of Half-Rectified Network Optimization
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Small nonlinearities in activation functions create bad local minima in neural networks
• Piecewise linear activations substantially shape the loss surfaces of neural networks
• Network size and weights size for memorization with two-layers neural networks
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently
• A Note on Connectivity of Sublevel Sets in Deep Learning
• Low-loss connection of weight vectors: distribution-based approaches