Three Learning Stages and Accuracy-Efficiency Tradeoff of Restricted Boltzmann Machines

Abstract

Restricted Boltzmann Machines (RBMs) offer a versatile architecture for unsupervised machine learning that can in principle approximate any target probability distribution with arbitrary accuracy. However, the RBM model is usually not directly accessible due to its computational complexity, and Markov-chain sampling is invoked to analyze the learned probability distribution. For training and eventual applications, it is thus desirable to have a sampler that is both accurate and efficient. We highlight that these two goals generally compete with each other and cannot be achieved simultaneously. More specifically, we identify and quantitatively characterize three regimes of RBM learning: independent learning, where the accuracy improves without losing efficiency; correlation learning, where higher accuracy entails lower efficiency; and degradation, where both accuracy and efficiency no longer improve or even deteriorate. These findings are based on numerical experiments and heuristic arguments.

arXiv comment14 pages, 4 figures (+ suppl. 10 pages, 9 figures)

References in corpus (20)

• Machine learning and the physical sciences
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• A high-bias, low-variance introduction to Machine Learning for physicists
• Generalized Gibbs ensemble in integrable lattice models
• Why does deep and cheap learning work so well?
• Efficient Representation of Quantum Many-body States with Deep Neural Networks
• Restricted-Boltzmann-Machine Learning for Solving Strongly Correlated Quantum Systems
• Fermionic neural-network states for ab-initio electronic structure
• A mathematical theory of semantic development in deep neural networks
• Estimation of effective temperatures in quantum annealers for sampling applications: A case study with possible applications in deep learning
• Mutual Information, Neural Networks and the Renormalization Group
• Quantum Machine Learning for Electronic Structure Calculations
• Latent Space Purification via Neural Density Operators
• Stochastic inference with spiking neurons in the high-conductance state
• The learnability scaling of quantum states: restricted Boltzmann machines
• Accelerated physical emulation of Bayesian inference in spiking neural networks
• Mode-Assisted Unsupervised Learning of Restricted Boltzmann Machines
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Spiking neuromorphic chip learns entangled quantum states
• Sampling scheme for neuromorphic simulation of entangled quantum systems

Cited by in corpus (2)

• CMOS + stochastic nanomagnets: heterogeneous computers for probabilistic inference and learning
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling