From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks

Abstract

We develop a statistical mechanical approach based on the replica method to study the design space of deep and wide neural networks constrained to meet a large number of training data. Specifically, we analyze the configuration space of the synaptic weights and neurons in the hidden layers in a simple feed-forward perceptron network for two scenarios: a setting with random inputs/outputs and a teacher-student setting. By increasing the strength of constraints,~i.e. increasing the number of training data, successive 2nd order glass transition (random inputs/outputs) or 2nd order crystalline transition (teacher-student setting) take place layer-by-layer starting next to the inputs/outputs boundaries going deeper into the bulk with the thickness of the solid phase growing logarithmically with the data size. This implies the typical storage capacity of the network grows exponentially fast with the depth. In a deep enough network, the central part remains in the liquid phase. We argue that in systems of finite width N, the weak bias field can remain in the center and plays the role of a symmetry-breaking field that connects the opposite sides of the system. The successive glass transitions bring about a hierarchical free-energy landscape with ultrametricity, which evolves in space: it is most complex close to the boundaries but becomes renormalized into progressively simpler ones in deeper layers. These observations provide clues to understand why deep neural networks operate efficiently. Finally, we present some numerical simulations of learning which reveal spatially heterogeneous glassy dynamics truncated by a finite width $N$ effect.

61 pages, 20 figures, revised version, to appear in SciPost Phys Core

References in corpus (11)

• Machine learning and the physical sciences
• Theoretical perspective on the glass transition and amorphous materials
• Fractal free energy landscapes in structural glasses
• Suppression of tunneling two-level systems in ultrastable glasses of indomethacin
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Exact theory of dense amorphous hard spheres in high dimension. I. The free energy
• Macroscopic Facilitation of Glassy Relaxation Kinetics: Ultra Stable Glass Films with Front-Like Thermal Response
• Analytic determination of dynamical and mosaic length scales in a Kac glass model
• Perspective: Gardner Physics in Amorphous Solids and Beyond
• Jamming in multilayer supervised learning models
• Comparing Dynamics: Deep Neural Networks versus Glassy Systems

Cited by in corpus (10)

• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Spatially heterogeneous learning by a deep student machine
• Proliferation of non-linear excitations in the piecewise-linear perceptron
• Dynamic Gardner crossover in a simple structural glass
• Data-driven effective model shows a liquid-like deep learning
• Discontinuous phase transition of feature detection in lateral predictive coding
• Variational Gaussian Approximation in Replica Analysis of Parametric Models
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation