Data-driven effective model shows a liquid-like deep learning

Abstract

The geometric structure of an optimization landscape is argued to be fundamentally important to support the success of deep neural network learning. A direct computation of the landscape beyond two layers is hard. Therefore, to capture the global view of the landscape, an interpretable model of the network-parameter (or weight) space must be established. However, the model is lacking so far. Furthermore, it remains unknown what the landscape looks like for deep networks of binary synapses, which plays a key role in robust and energy efficient neuromorphic computation. Here, we propose a statistical mechanics framework by directly building a least structured model of the high-dimensional weight space, considering realistic structured data, stochastic gradient descent training, and the computational depth of neural networks. We also consider whether the number of network parameters outnumbers the number of supplied training data, namely, over- or under-parametrization. Our least structured model reveals that the weight spaces of the under-parametrization and over-parameterization cases belong to the same class, in the sense that these weight spaces are well-connected without any hierarchical clustering structure. In contrast, the shallow-network has a broken weight space, characterized by a discontinuous phase transition, thereby clarifying the benefit of depth in deep learning from the angle of high dimensional geometry. Our effective model also reveals that inside a deep network, there exists a liquid-like central part of the architecture in the sense that the weights in this part behave as randomly as possible, providing algorithmic implications. Our data-driven model thus provides a statistical mechanics insight about why deep learning is unreasonably effective in terms of the high-dimensional weight space, and how deep networks are different from shallow ones.

arXiv comment43 pages, 19 figures, 2nd revision for the journal

References in corpus (23)

• Machine learning and the physical sciences
• Weak pairwise correlations imply strongly correlated network states in a neural population
• Reconciling modern machine learning practice and the bias-variance trade-off
• The Bethe lattice spin glass revisited
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• High-dimensional Ising model selection using ${\ell }_1$-regularized logistic regression
• The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence
• Inverse Ising inference using all the data
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Perspective: Energy Landscapes for Machine Learning
• Shaping the learning landscape in neural networks around wide flat minima
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Origin of the computational hardness for learning with binary synapses
• Comparing Dynamics: Deep Neural Networks versus Glassy Systems
• Entropy landscape of solutions in the binary perceptron problem
• Geometry of energy landscapes and the optimizability of deep neural networks
• Learning credit assignment
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• Clustering of neural codewords revealed by a first-order phase transition
• Random active path model of deep neural networks with diluted binary synapses
• Counting solutions from finite samplings
• Theory of population coupling and applications to describe high order correlations in large populations of interacting neurons

Cited by in corpus (3)

• Unveiling the structure of wide flat minima in neural networks
• Eight challenges in developing theory of intelligence
• Spatially heterogeneous learning by a deep student machine