Emergence of Compositional Representations in Restricted Boltzmann Machines

Abstract

Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits dataset MNIST.

Supplementary material available at the authors' webpage

References in corpus (1)

• Extensive load in multitasking associative networks

Cited by in corpus (54)

• Machine learning and the physical sciences
• Deep Learning in Spiking Neural Networks
• A high-bias, low-variance introduction to Machine Learning for physicists
• Deep learning for comprehensive forecasting of Alzheimer's Disease progression
• Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
• Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Analytic continuation via domain-knowledge free machine learning
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Mean-field inference methods for neural networks
• Non-Convex Multi-species Hopfield models
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Dreaming neural networks: rigorous results
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Statistical Physics and Representations in Real and Artificial Neural Networks
• A developmental approach for training deep belief networks
• Unsupervised hierarchical clustering using the learning dynamics of RBMs
• Quantifying Relevance in Learning and Inference
• Approximate Survey Propagation for Statistical Inference
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Restricted Boltzmann Machines and Matrix Product States of 1D Translational Invariant Stabilizer Codes
• Minimal model of permutation symmetry in unsupervised learning
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Disentangling representations in Restricted Boltzmann Machines without adversaries
• Inferring effective couplings with Restricted Boltzmann Machines
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• Machine learning phases and criticalities without using real data for training
• Generalized hetero-associative neural networks
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• Parallel Learning by Multitasking Neural Networks
• The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Gaussian-Spherical Restricted Boltzmann Machines
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• Role of zero synapses in unsupervised feature learning
• Interpretable machine learning of amino acid patterns in proteins: a statistical ensemble approach
• Machine learning phases of an Abelian gauge theory
• The Relativistic Hopfield network: rigorous results
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• On the role of non-linear latent features in bipartite generative neural networks
• Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
• A universal neural network for learning phases and criticalities
• Self-regularizing restricted Boltzmann machines
• Oscillatory evolution of collective behavior in evolutionary games played with reinforcement learning
• Maximal Relevance and Optimal Learning Machines
• Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling
• GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data
• Decreasing the size of the Restricted Boltzmann machine