A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

Abstract

Restricted Boltzmann machines (RBMs) are energy-based neural-networks which are commonly used as the building blocks for deep architectures neural architectures. In this work, we derive a deterministic framework for the training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer (TAP) mean-field approximation of widely-connected systems with weak interactions coming from spin-glass theory. While the TAP approach has been extensively studied for fully-visible binary spin systems, our construction is generalized to latent-variable models, as well as to arbitrarily distributed real-valued spin systems with bounded support. In our numerical experiments, we demonstrate the effective deterministic training of our proposed models and are able to show interesting features of unsupervised learning which could not be directly observed with sampling. Additionally, we demonstrate how to utilize our TAP-based framework for leveraging trained RBMs as joint priors in denoising problems.

References in corpus (8)

• Improved contact prediction in proteins: Using pseudolikelihoods to infer Potts models
• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Small-correlation expansions for the inverse Ising problem
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Variational perturbation and extended Plefka approaches to dynamics on random networks: the case of the kinetic Ising model

Cited by in corpus (14)

• Machine learning and the physical sciences
• Restricted Boltzmann Machine, recent advances and mean-field theory
• High-temperature Expansions and Message Passing Algorithms
• Mean-field inference methods for neural networks
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• Quantum neural networks to simulate many-body quantum systems
• Unsupervised hierarchical clustering using the learning dynamics of RBMs
• Memory-free dynamics for the TAP equations of Ising models with arbitrary rotation invariant ensembles of random coupling matrices
• Inferring effective couplings with Restricted Boltzmann Machines
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• Privacy-preserving machine learning with tensor networks
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Analysis of Bayesian Inference Algorithms by the Dynamical Functional Approach
• Neuromodulation via Krotov-Hopfield Improves Accuracy and Robustness of RBMs