The Gaussian equivalence of generative models for learning with shallow neural networks

Abstract

Understanding the impact of data structure on the computational tractability of learning is a key challenge for the theory of neural networks. Many theoretical works do not explicitly model training data, or assume that inputs are drawn component-wise independently from some simple probability distribution. Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models. This is possible due to a Gaussian equivalence stating that the key metrics of interest, such as the training and test errors, can be fully captured by an appropriately chosen Gaussian model. We provide three strands of rigorous, analytical and numerical evidence corroborating this equivalence. First, we establish rigorous conditions for the Gaussian equivalence to hold in the case of single-layer generative models, as well as deterministic rates for convergence in distribution. Second, we leverage this equivalence to derive a closed set of equations describing the generalisation performance of two widely studied machine learning problems: two-layer neural networks trained using one-pass stochastic gradient descent, and full-batch pre-learned features or kernel methods. Finally, we perform experiments demonstrating how our theory applies to deep, pre-trained generative models. These results open a viable path to the theoretical study of machine learning models with realistic data.

arXiv commentThe accompanying code for this paper is available at https://github.com/sgoldt/gaussian-equiv-2layer

References in corpus (27)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• NICE: Non-linear Independent Components Estimation
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Variational Inference with Normalizing Flows
• Normalizing Flows: An Introduction and Review of Current Methods
• Deep Neural Networks as Gaussian Processes
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Statistical physics of inference: Thresholds and algorithms
• Normalizing Flows for Probabilistic Modeling and Inference
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Intrinsic dimension of data representations in deep neural networks
• Entropy and mutual information in models of deep neural networks
• Classification and Geometry of General Perceptual Manifolds
• Recovery Guarantees for One-hidden-layer Neural Networks
• Mean-field message-passing equations in the Hopfield model and its generalizations
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Universality Laws for High-Dimensional Learning with Random Features
• A Precise Performance Analysis of Learning with Random Features
• Counting the learnable functions of structured data
• A Probabilistic Framework for Deep Learning
• Learning Data Manifolds with a Cutting Plane Method
• Deep Learning and Hierarchal Generative Models

Cited by in corpus (21)

• Generalisation error in learning with random features and the hidden manifold model
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Probing transfer learning with a model of synthetic correlated datasets
• Universality Laws for High-Dimensional Learning with Random Features
• Gaussian Universality of Perceptrons with Random Labels
• On the Inherent Regularization Effects of Noise Injection During Training
• On the interplay between data structure and loss function in classification problems
• Theoretical characterization of uncertainty in high-dimensional linear classification
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• Observing Schrödinger's Cat with Artificial Intelligence: Emergent Classicality from Information Bottleneck
• Fast Approximation of the Sliced-Wasserstein Distance Using Concentration of Random Projections
• The impact of memory on learning sequence-to-sequence tasks
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
• Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
• Appearance of Random Matrix Theory in Deep Learning
• A Dynamical Central Limit Theorem for Shallow Neural Networks
• Covariate Shift in High-Dimensional Random Feature Regression