Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization

Abstract

The success of deep learning in many real-world tasks has triggered an intense effort to understand the power and limitations of deep learning in the training and generalization of complex tasks, so far with limited progress. In this work, we study the statistical mechanics of learning in Deep Linear Neural Networks (DLNNs) in which the input-output function of an individual unit is linear. Despite the linearity of the units, learning in DLNNs is nonlinear, hence studying its properties reveals some of the features of nonlinear Deep Neural Networks (DNNs). Importantly, we solve exactly the network properties following supervised learning using an equilibrium Gibbs distribution in the weight space. To do this, we introduce the Back-Propagating Kernel Renormalization (BPKR), which allows for the incremental integration of the network weights starting from the network output layer and progressing backward until the first layer's weights are integrated out. This procedure allows us to evaluate important network properties, such as its generalization error, the role of network width and depth, the impact of the size of the training set, and the effects of weight regularization and learning stochasticity. BPKR does not assume specific statistics of the input or the task's output. Furthermore, by performing partial integration of the layers, the BPKR allows us to compute the properties of the neural representations across the different hidden layers. We have proposed an extension of the BPKR to nonlinear DNNs with ReLU. Surprisingly, our numerical simulations reveal that despite the nonlinearity, the predictions of our theory are largely shared by ReLU networks in a wide regime of parameters. Our work is the first exact statistical mechanical study of learning in a family of DNNs, and the first successful theory of learning through successive integration of DoFs in the learned weight space.

arXiv commentmain text 31 pages, 15 figures

References in corpus (10)

• Machine learning and the physical sciences
• Kernel methods in machine learning
• Reconciling modern machine learning practice and the bias-variance trade-off
• The Renormalization Group and Singular Perturbations: Multiple-Scales, Boundary Layers and Reductive Perturbation Theory
• A mathematical theory of semantic development in deep neural networks
• Neural Network Renormalization Group
• Classification and Geometry of General Perceptual Manifolds
• Perspective: Energy Landscapes for Machine Learning
• Statistical mechanics of complex neural systems and high dimensional data
• Geometry of energy landscapes and the optimizability of deep neural networks

Cited by in corpus (15)

• Asymptotics of representation learning in finite Bayesian neural networks
• Contrasting random and learned features in deep Bayesian linear regression
• Unified field theoretical approach to deep and recurrent neuronal networks
• Gaussian Universality of Perceptrons with Random Labels
• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• Exact full-RSB SAT/UNSAT transition in infinitely wide two-layer neural networks
• Error Scaling Laws for Kernel Classification under Source and Capacity Conditions
• Bayesian RG Flow in Neural Network Field Theories
• Wilsonian Renormalization of Neural Network Gaussian Processes
• Meta-Learning Strategies through Value Maximization in Neural Networks
• Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis
• Bayesian inference with finitely wide neural networks
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation