'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space

Abstract

Distributions of data or sensory stimuli often enjoy underlying invariances. How and to what extent those symmetries are captured by unsupervised learning methods is a relevant question in machine learning and in computational neuroscience. We study here, through a combination of numerical and analytical tools, the learning dynamics of Restricted Boltzmann Machines (RBM), a neural network paradigm for representation learning. As learning proceeds from a random configuration of the network weights, we show the existence of, and characterize a symmetry-breaking phenomenon, in which the latent variables acquire receptive fields focusing on limited parts of the invariant manifold supporting the data. The symmetry is restored at large learning times through the diffusion of the receptive field over the invariant manifold; hence, the RBM effectively spans a continuous attractor in the space of network weights. This symmetry-breaking phenomenon takes place only if the amount of data available for training exceeds some critical value, depending on the network size and the intensity of symmetry-induced correlations in the data; below this 'retarded-learning' threshold, the network weights are essentially noisy and overfit the data.

References in corpus (6)

• Mutual Information, Neural Networks and the Renormalization Group
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Nonlinear Hebbian learning as a unifying principle in receptive field formation
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks

Cited by in corpus (9)

• Restricted Boltzmann Machine, recent advances and mean-field theory
• Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
• Data-driven emergence of convolutional structure in neural networks
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Disentangling representations in Restricted Boltzmann Machines without adversaries
• On the role of non-linear latent features in bipartite generative neural networks
• Emergence of global receptive fields capturing multipartite quantum correlations
• Tradeoff of generalization error in unsupervised learning