Taxonomizing local versus global structure in neural network loss landscapes

Abstract

Viewing neural network models in terms of their loss landscapes has a long history in the statistical mechanics approach to learning, and in recent years it has received attention within machine learning proper. Among other things, local metrics (such as the smoothness of the loss landscape) have been shown to correlate with global properties of the model (such as good generalization performance). Here, we perform a detailed empirical analysis of the loss landscape structure of thousands of neural network models, systematically varying learning tasks, model architectures, and/or quantity/quality of data. By considering a range of metrics that attempt to capture different aspects of the loss landscape, we demonstrate that the best test accuracy is obtained when: the loss landscape is globally well-connected; ensembles of trained models are more similar to each other; and models converge to locally smooth regions. We also show that globally poorly-connected landscapes can arise when models are small or when they are trained to lower quality data; and that, if the loss landscape is globally poorly-connected, then training to zero loss can actually lead to worse test accuracy. Our detailed empirical results shed light on phases of learning (and consequent double descent behavior), fundamental versus incidental determinants of good generalization, the role of load-like and temperature-like parameters in the learning process, different influences on the loss landscape from model and data, and the relationships between local and global metrics, all topics of recent interest.

References in corpus (21)

• The Loss Surfaces of Multilayer Networks
• Benign Overfitting in Linear Regression
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Similarity of Neural Network Representations Revisited
• Qualitatively characterizing neural network optimization problems
• Gradient Descent Happens in a Tiny Subspace
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• The Global Landscape of Neural Networks: An Overview
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• Multiple Descent: Design Your Own Generalization Curve
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Emergent properties of the local geometry of neural loss landscapes
• Understanding Why Neural Networks Generalize Well Through GSNR of Parameters
• Improved guarantees and a multiple-descent curve for Column Subset Selection and the Nyström method
• A walk in the statistical mechanical formulation of neural networks
• Flatness is a False Friend
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics