Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets

Abstract

Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima -- at least those discovered by gradient-based optimization -- turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segments. We give mathematical explanations for this phenomenon, assuming generic properties (such as dropout stability and noise stability) of well-trained deep nets, which have previously been identified as part of understanding the generalization properties of deep nets. Our explanation holds for realistic multilayer nets, and experiments are presented to verify the theory.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• On the importance of single directions for generalization
• Essentially No Barriers in Neural Network Energy Landscape
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• On the loss landscape of a class of deep neural networks with no bad local valleys
• Understanding the Loss Surface of Neural Networks for Binary Classification
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes

Cited by in corpus (16)

• On the Opportunities and Risks of Foundation Models
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• The Global Landscape of Neural Networks: An Overview
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walks
• Linear Mode Connectivity in Multitask and Continual Learning
• Taxonomizing local versus global structure in neural network loss landscapes
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Towards Deep Learning Models Resistant to Large Perturbations
• When Are Solutions Connected in Deep Networks?
• Tighter Sparse Approximation Bounds for ReLU Neural Networks
• Learning the mapping $\mathbf{x}\mapsto {\sum }_{i=1}^d{x}_i^2$: the cost of finding the needle in a haystack
• WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points