Mean-field inference methods for neural networks

Abstract

Machine learning algorithms relying on deep neural networks recently allowed a great leap forward in artificial intelligence. Despite the popularity of their applications, the efficiency of these algorithms remains largely unexplained from a theoretical point of view. The mathematical description of learning problems involves very large collections of interacting random variables, difficult to handle analytically as well as numerically. This complexity is precisely the object of study of statistical physics. Its mission, originally pointed towards natural systems, is to understand how macroscopic behaviors arise from microscopic laws. Mean-field methods are one type of approximation strategy developed in this view. We review a selection of classical mean-field methods and recent progress relevant for inference in neural networks. In particular, we remind the principles of derivations of high-temperature expansions, the replica method and message passing algorithms, highlighting their equivalences and complementarities. We also provide references for past and current directions of research on neural networks relying on mean-field methods.

References in corpus (48)

• Variational Inference: A Review for Statisticians
• Message Passing Algorithms for Compressed Sensing
• Machine learning and the physical sciences
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Analytical Solution of the Off-Equilibrium Dynamics of a Long Range Spin-Glass Model
• The Bethe lattice spin glass revisited
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Spin-Glass Theory for Pedestrians
• Statistical physics of inference: Thresholds and algorithms
• Inverse statistical problems: from the inverse Ising problem to data science
• Three Factors Influencing Minima in SGD
• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Entropy and mutual information in models of deep neural networks
• Classification and Geometry of General Perceptual Manifolds
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• The Nishimori line and Bayesian Statistics
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Phase transitions and sample complexity in Bayes-optimal matrix factorization
• Mean-Field Equations for Spin Models with Orthogonal Interaction Matrices
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications
• Tractable approximations for probabilistic models: The adaptive TAP mean field approach
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Statistical mechanics of complex neural systems and high dimensional data
• Origin of the computational hardness for learning with binary synapses
• Phase transitions in Restricted Boltzmann Machines with generic priors
• A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels
• High-temperature Expansions and Message Passing Algorithms
• Entropy landscape of solutions in the binary perceptron problem
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Perceptron capacity revisited: classification ability for correlated patterns
• Approximate Message Passing with Restricted Boltzmann Machine Priors
• Glassy nature of the hard phase in inference problems
• Memory-free dynamics for the TAP equations of Ising models with arbitrary rotation invariant ensembles of random coupling matrices
• Approximate Survey Propagation for Statistical Inference
• On the role of synaptic stochasticity in training low-precision neural networks
• Exploring the Function Space of Deep-Learning Machines
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• Phase Retrieval Under a Generative Prior
• Learning of correlated patterns by simple perceptrons
• Online compressed sensing
• Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
• Blind calibration for compressed sensing: State evolution and an online algorithm

Cited by in corpus (9)

• Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation
• Roadmap on machine learning glassy dynamics
• Dynamics of coordinate ascent variational inference: A case study in 2D Ising models
• Deep learning via message passing algorithms based on belief propagation
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• Partial local entropy and anisotropy in deep weight spaces
• Appearance of Random Matrix Theory in Deep Learning
• Bayesian inference with finitely wide neural networks