Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses

Abstract

Revealing hidden features in unlabeled data is called unsupervised feature learning, which plays an important role in pretraining a deep neural network. Here we provide a statistical mechanics analysis of the unsupervised learning in a restricted Boltzmann machine with binary synapses. A message passing equation to infer the hidden feature is derived, and furthermore, variants of this equation are analyzed. A statistical analysis by replica theory describes the thermodynamic properties of the model. Our analysis confirms an entropy crisis preceding the non-convergence of the message passing equation, suggesting a discontinuous phase transition as a key characteristic of the restricted Boltzmann machine. Continuous phase transition is also confirmed depending on the embedded feature strength in the data. The mean-field result under the replica symmetric assumption agrees with that obtained by running message passing algorithms on single instances of finite sizes. Interestingly, in an approximate Hopfield model, the entropy crisis is absent, and a continuous phase transition is observed instead. We also develop an iterative equation to infer the hyper-parameter (temperature) hidden in the data, which in physics corresponds to iteratively imposing Nishimori condition. Our study provides insights towards understanding the thermodynamic properties of the restricted Boltzmann machine learning, and moreover important theoretical basis to build simplified deep networks.

24 pages, 9 figures, results added

References in corpus (4)

• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition

Cited by in corpus (22)

• A high-bias, low-variance introduction to Machine Learning for physicists
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Mean-field inference methods for neural networks
• Non-Convex Multi-species Hopfield models
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Statistical physics of unsupervised learning with prior knowledge in neural networks
• Generalized hetero-associative neural networks
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Parallel Learning by Multitasking Neural Networks
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Analyticity of the energy in an Ising spin glass with correlated disorder
• Eigenvalue spectrum of neural networks with arbitrary Hebbian length
• Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems
• Saddle Hierarchy in Dense Associative Memory
• Supervised and Unsupervised protocols for hetero-associative neural networks