Advanced Mean Field Theory of Restricted Boltzmann Machine
arXiv:1502.00186 · doi:10.1103/PhysRevE.91.050101
Abstract
Learning in restricted Boltzmann machine is typically hard due to the computation of gradients of log-likelihood function. To describe the network state statistics of the restricted Boltzmann machine, we develop an advanced mean field theory based on the Bethe approximation. Our theory provides an efficient message passing based method that evaluates not only the partition function (free energy) but also its gradients without requiring statistical sampling. The results are compared with those obtained by the computationally expensive sampling based method.
arXiv comment5 pages, 4 figures, accepted by Phys Rev E (Rapid Communication)
References in corpus (5)
Cited by in corpus (27)
- Estimation of effective temperatures in quantum annealers for sampling applications: A case study with possible applications in deep learning
- Mean-field message-passing equations in the Hopfield model and its generalizations
- Machine Learning for Condensed Matter Physics
- Restricted Boltzmann Machine, recent advances and mean-field theory
- Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow
- Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
- Spectral Dynamics of Learning Restricted Boltzmann Machines
- Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
- Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
- Mechanisms of dimensionality reduction and decorrelation in deep neural networks
- Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines
- A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
- Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
- Statistical physics of unsupervised learning with prior knowledge in neural networks
- Minimal model of permutation symmetry in unsupervised learning
- Boltzmann machines as two-dimensional tensor networks
- Mean-Field Inference in Gaussian Restricted Boltzmann Machine
- Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
- Space of Functions Computed by Deep-Layered Machines
- Singular-Value-Decomposition Analysis of Associative Memory in a Neural Network
- Role of zero synapses in unsupervised feature learning
- Thermodynamics of the Ising model encoded in restricted Boltzmann machines
- Variational mean-field theory for training restricted Boltzmann machines with binary synapses
- Circumventing spin glass traps by microcanonical spontaneous symmetry breaking
- Efficient Evaluation of the Partition Function of RBMs with Annealed Importance Sampling
- Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
- Exact training of Restricted Boltzmann machines on intrinsically low dimensional data