Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network

Abstract

We analytically derive the geometrical structure of the weight space in multilayer neural networks (MLN), in terms of the volumes of couplings associated to the internal representations of the training set. Focusing on the parity and committee machines, we deduce their learning and generalization capabilities both reinterpreting some known properties and finding new exact results. The relationship between our approach and information theory as well as the Mitchison--Durbin calculation is established. Our results are exact in the limit of a large number of hidden units, showing that MLN are a class of exactly solvable models with a simple interpretation of replica symmetry breaking.

12 pages, 1 compressed ps figure (uufile), RevTeX file

Cited by in corpus (42)

• Structure of Growing Networks: Exact Solution of the Barabasi--Albert's Model
• Evolution of reference networks with aging
• Scaling Behaviour of Developing and Decaying Networks
• Statistical mechanics of the random K-SAT model
• Coloring random graphs
• Statistical Mechanics of Support Vector Networks
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Polynomial iterative algorithms for coloring and analyzing random graphs
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Shaping the learning landscape in neural networks around wide flat minima
• Statistical mechanics of complex neural systems and high dimensional data
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Solving satisfiability problems by fluctuations: The dynamics of stochastic local search algorithms
• Random Graph Coloring - a Statistical Physics Approach
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Mean-field inference methods for neural networks
• The committee machine: Computational to statistical gaps in learning a two-layers neural network
• Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron
• Replica symmetry breaking in dense neural networks
• Analytical and Numerical Study of Internal Representations in Multilayer Neural Networks with Binary Weights
• Activation function dependence of the storage capacity of treelike neural networks
• Multifractal Analysis of the Coupling Space of Feed-Forward Neural Networks
• Teacher-student learning for a binary perceptron with quantum fluctuations
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• On the VC-dimension of neural networks with binary weights
• Generalization from correlated sets of patterns in the perceptron
• Multilayer neural networks with extensively many hidden units
• Phase transitions in the generalization behaviour of multilayer perceptrons: II. The influence of noise
• Correlation of internal representations in feed-forward neural networks
• Multifractality and percolation in the coupling space of perceptrons
• Spatially heterogeneous learning by a deep student machine
• The Copycat Perceptron: Smashing Barriers Through Collective Learning
• Storage capacity of correlated perceptrons
• Correlations between hidden units in multilayer neural networks and replica symmetry breaking
• A multifractal phase-space analysis of perceptrons with biased patterns
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Statistical Physics of Feedforward Neural Networks
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• How isotropic kernels perform on simple invariants
• Multifractal analysis of perceptron learning with errors
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Evolving Network With Different Edges