Modelling the influence of data structure on learning in neural networks: the hidden manifold model

Abstract

Understanding the reasons for the success of deep neural networks trained using stochastic gradient-based methods is a key open problem for the nascent theory of deep learning. The types of data where these networks are most successful, such as images or sequences of speech, are characterised by intricate correlations. Yet, most theoretical work on neural networks does not explicitly model training data, or assumes that elements of each data sample are drawn independently from some factorised probability distribution. These approaches are thus by construction blind to the correlation structure of real-world data sets and their impact on learning in neural networks. Here, we introduce a generative model for structured data sets that we call the hidden manifold model (HMM). The idea is to construct high-dimensional inputs that lie on a lower-dimensional manifold, with labels that depend only on their position within this manifold, akin to a single layer decoder or generator in a generative adversarial network. We demonstrate that learning of the hidden manifold model is amenable to an analytical treatment by proving a "Gaussian Equivalence Property" (GEP), and we use the GEP to show how the dynamics of two-layer neural networks trained using one-pass stochastic gradient descent is captured by a set of integro-differential equations that track the performance of the network at all times. This permits us to analyse in detail how a neural network learns functions of increasing complexity during training, how its performance depends on its size and how it is impacted by parameters such as the learning rate or the dimension of the hidden manifold.

References in corpus (9)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Statistical physics of inference: Thresholds and algorithms
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Deterministic equivalents for certain functionals of large random matrices
• Classification and Geometry of General Perceptual Manifolds
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Dynamics of Learning with Restricted Training Sets I: General Theory
• Counting the learnable functions of structured data
• Learning Data Manifolds with a Cutting Plane Method

Cited by in corpus (49)

• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Perspectives on adaptive dynamical systems
• Measurement-induced criticality as a data-structure transition
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Unsupervised learning universal critical behavior via the intrinsic dimension
• Quantum Kernel Methods under Scrutiny: A Benchmarking Study
• Dimensionality reduction of local structure in glassy binary mixtures
• Mapping of attention mechanisms to a generalized Potts model
• Intrinsic dimension of path integrals: data mining quantum criticality and emergent simplicity
• Learning through atypical "phase transitions" in overparameterized neural networks
• Contrasting random and learned features in deep Bayesian linear regression
• Probing transfer learning with a model of synthetic correlated datasets
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Activation function dependence of the storage capacity of treelike neural networks
• Learning curves for the multi-class teacher-student perceptron
• Gaussian Universality of Perceptrons with Random Labels
• Linear Classification of Neural Manifolds with Correlated Variability
• Deep neural networks have an inbuilt Occam's razor
• Deep learning via message passing algorithms based on belief propagation
• Supervised Learning in the Presence of Concept Drift: A modelling framework
• Decomposing neural networks as mappings of correlation functions
• Investigating Berezinskii-Kosterlitz-Thouless phase transitions in Kagome spin ice by quantifying Monte Carlo process: Distribution of Hamming distances
• Random features and polynomial rules
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• Spatially heterogeneous learning by a deep student machine
• Symmetry & critical points for a model shallow neural network
• Theoretical characterization of uncertainty in high-dimensional linear classification
• Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines
• On the similarity of bandwidth-tuned quantum kernels and classical kernels
• Optimal Learning with Excitatory and Inhibitory synapses
• The impact of memory on learning sequence-to-sequence tasks
• The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• Analysis of Diffusion Models for Manifold Data
• Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens
• Noise Balance and Stationary Distribution of Stochastic Gradient Descent
• Dynamics of Meta-learning Representation in the Teacher-student Scenario
• Generative random latent features models and statistics of natural images
• Summary statistics of learning link changing neural representations to behavior
• On the role of non-linear latent features in bipartite generative neural networks
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Daydreaming algorithm for Biased Patterns
• Category learning in deep neural networks: Information content and geometry of internal representations
• Critical properties of the SAT/UNSAT transitions in the classification problem of structured data
• Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networks