Self-regularizing restricted Boltzmann machines

Abstract

Focusing on the grand-canonical extension of the ordinary restricted Boltzmann machine, we suggest an energy-based model for feature extraction that uses a layer of hidden units with varying size. By an appropriate choice of the chemical potential and given a sufficiently large number of hidden resources the generative model is able to efficiently deduce the optimal number of hidden units required to learn the target data with exceedingly small generalization error. The formal simplicity of the grand-canonical ensemble combined with a rapidly converging ansatz in mean-field theory enable us to recycle well-established numerical algothhtims during training, like contrastive divergence, with only minor changes. As a proof of principle and to demonstrate the novel features of grand-canonical Boltzmann machines, we train our generative models on data from the Ising theory and MNIST.

28 pages, 11 figures

References in corpus (24)

• On the difficulty of training Recurrent Neural Networks
• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Boosted Decision Trees as an Alternative to Artificial Neural Networks for Particle Identification
• Quantum Entanglement in Neural Network States
• Learning Thermodynamics with Boltzmann Machines
• Mutual Information, Neural Networks and the Renormalization Group
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Restricted Boltzmann Machines for Quantum States with Nonabelian or Anyonic Symmetries
• Deep Learning and AdS/CFT
• Machine Learning Spatial Geometry from Entanglement Features
• Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow
• Deep Learning the Ising Model Near Criticality
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Holography as deep learning
• Thermodynamics and Feature Extraction by Machine Learning
• Machine learning determination of dynamical parameters: The Ising model case
• The Calabi-Yau Landscape: from Geometry, to Physics, to Machine-Learning
• Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design
• Geometrization of deep networks for the interpretability of deep learning systems
• Regularization for Unsupervised Deep Neural Nets
• Variational approach to unsupervised learning
• An Infinite Restricted Boltzmann Machine