Spatially heterogeneous learning by a deep student machine

Abstract

Deep neural networks (DNN) with a huge number of adjustable parameters remain largely black boxes. To shed light on the hidden layers of DNN, we study supervised learning by a DNN of width $N$ and depth $L$ consisting of $NL$ perceptrons with $c$ inputs by a statistical mechanics approach called the teacher-student setting. We consider an ensemble of student machines that exactly reproduce $M$ sets of $N$ dimensional input/output relations provided by a teacher machine. We show that the problem becomes exactly solvable in what we call as 'dense limit': $N\gg c\gg 1$ and $M\gg 1$ with fixed $\alpha =M/c$ using the replica method developed in (H. Yoshino, (2020)). We also study the model numerically performing simple greedy MC simulations. Simulations reveal that learning by the DNN is quite heterogeneous in the network space: configurations of the teacher and the student machines are more correlated within the layers closer to the input/output boundaries while the central region remains much less correlated due to the over-parametrization in qualitative agreement with the theoretical prediction. We evaluate the generalization-error of the DNN with various depth $L$ both theoretically and numerically. Remarkably both the theory and simulation suggest generalization-ability of the student machines, which are only weakly correlated with the teacher in the center, does not vanish even in the deep limit $L\gg 1$ where the system becomes heavily over-parametrized. We also consider the impact of effective dimension $D(\le N)$ of data by incorporating the hidden manifold model (S. Goldt et. al., (2020)) into our model. The theory implies that the loop corrections to the dense limit become enhanced by either decreasing the width $N$ or decreasing the effective dimension $D$ of the data. Simulation suggests both lead to significant improvements in generalization-ability.

arXiv comment41 page, 19 figures (revised version)

References in corpus (13)

• Machine learning and the physical sciences
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Statistical physics of inference: Thresholds and algorithms
• Recipes for metastable states in Spin Glasses
• Scaling description of generalization with number of parameters in deep learning
• The Nishimori line and Bayesian Statistics
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Chiral-glass transition and replica symmetry breaking of a three-dimensional Heisenberg spin glass
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Glassy dynamics as a melting process (On melting dynamics and the glass transition, Part II)
• Glassy aspects of melting dynamics (On melting dynamics and the glass transition, Part I)
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• Data-driven effective model shows a liquid-like deep learning

Cited by in corpus (5)

• A deep neural network approach to solve the Dirac equation
• Variational Gaussian Approximation in Replica Analysis of Parametric Models
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation