Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy

Abstract

Restricted Boltzmann machines are undirected neural networks which have been shown to be effective in many applications, including serving as initializations for training deep multi-layer neural networks. One of the main reasons for their success is the existence of efficient and practical stochastic algorithms, such as contrastive divergence, for unsupervised training. We propose an alternative deterministic iterative procedure based on an improved mean field method from statistical physics known as the Thouless-Anderson-Palmer approach. We demonstrate that our algorithm provides performance equal to, and sometimes superior to, persistent contrastive divergence, while also providing a clear and easy to evaluate objective function. We believe that this strategy can be easily generalized to other models as well as to more accurate higher-order approximations, paving the way for systematic improvements in training Boltzmann machines with hidden units.

8 pages, 7 figures, demo online at http://www.lps.ens.fr/~krzakala/WASP.html

References in corpus (2)

• Scikit-learn: Machine Learning in Python
• Joint Training Deep Boltzmann Machines for Classification

Cited by in corpus (5)

• Restricted Boltzmann Machine, recent advances and mean-field theory
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Boltzmann machines as two-dimensional tensor networks
• Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines
• From Boltzmann Machines to Neural Networks and Back Again