Annealing and replica-symmetry in Deep Boltzmann Machines

Abstract

In this paper we study the properties of the quenched pressure of a multi-layer spin-glass model (a deep Boltzmann Machine in artificial intelligence jargon) whose pairwise interactions are allowed between spins lying in adjacent layers and not inside the same layer nor among layers at distance larger than one. We prove a theorem that bounds the quenched pressure of such a K-layer machine in terms of K Sherrington-Kirkpatrick spin glasses and use it to investigate its annealed region. The replica-symmetric approximation of the quenched pressure is identified and its relation to the annealed one is considered. The paper also presents some observation on the model's architectural structure related to machine learning. Since escaping the annealed region is mandatory for a meaningful training, by squeezing such region we obtain thermodynamical constraints on the form factors. Remarkably, its optimal escape is achieved by requiring the last layer to scale sub-linearly in the network size.

arXiv comment15 pages, 1 figure. To appear in the Special Issue dedicated to celebrating Joel Lebowitz

References in corpus (12)

• Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications
• Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
• The Thermodynamic Limit in Mean Field Spin Glass Models
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Mean-field message-passing equations in the Hopfield model and its generalizations
• The free energy in a multi-species Sherrington-Kirkpatrick model
• Equilibrium statistical mechanics of bipartite spin systems
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Free energy and complexity of spherical bipartite models
• High-Dimensional Inference with the generalized Hopfield Model: Principal Component Analysis and Corrections
• Non-Convex Multi-species Hopfield models

Cited by in corpus (14)

• Quantum field-theoretic machine learning
• Medical Imaging with Deep Learning for COVID- 19 Diagnosis: A Comprehensive Review
• Deep Boltzmann machines: rigorous results at arbitrary depth
• Free energy in multi-species mixed $p$-spin spherical models
• The solution of the deep Boltzmann machine on the Nishimori line
• Dense Hebbian neural networks: a replica symmetric picture of supervised learning
• The multi-species mean-field spin-glass on the Nishimori line
• Fluctuation results for Multi-species Sherrington-Kirkpatrick model in the replica symmetric regime
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• On the free energy of vector spin glasses with non-convex interactions
• Dynamics of one-dimensional spin models under the line-graph operator
• Pattern recognition in Deep Boltzmann machines
• Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey