Variational mean-field theory for training restricted Boltzmann machines with binary synapses

Abstract

Unsupervised learning requiring only raw data is not only a fundamental function of the cerebral cortex, but also a foundation for a next generation of artificial neural networks. However, a unified theoretical framework to treat sensory inputs, synapses and neural activity together is still lacking. The computational obstacle originates from the discrete nature of synapses, and complex interactions among these three essential elements of learning. Here, we propose a variational mean-field theory in which the distribution of synaptic weights is considered. The unsupervised learning can then be decomposed into two intertwined steps: a maximization step is carried out as a gradient ascent of the lower-bound on the data log-likelihood, in which the synaptic weight distribution is determined by updating variational parameters, and an expectation step is carried out as a message passing procedure on an equivalent or dual neural network whose parameter is specified by the variational parameters of the weight distribution. Therefore, our framework provides insights on how data (or sensory inputs), synapses and neural activities interact with each other to achieve the goal of extracting statistical regularities in sensory inputs. This variational framework is verified in restricted Boltzmann machines with planted synaptic weights and handwritten-digits learning.

arXiv comment9 pages, 2 figures, a mean-field framework proposed for unsupervised learning in RBM with discrete synapses, which was previously out of reach

References in corpus (15)

• Variational Inference: A Review for Statisticians
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• A high-bias, low-variance introduction to Machine Learning for physicists
• The Bethe lattice spin glass revisited
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• On the role of synaptic stochasticity in training low-precision neural networks
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Minimal model of permutation symmetry in unsupervised learning
• Statistical physics of unsupervised learning with prior knowledge in neural networks
• Role of zero synapses in unsupervised feature learning

Cited by in corpus (4)

• Restricted Boltzmann Machine, recent advances and mean-field theory
• Learning credit assignment
• Statistical mechanics of continual learning: variational principle and mean-field potential
• Training Restricted Boltzmann Machines with Binary Synapses using the Bayesian Learning Rule