Minimal model of permutation symmetry in unsupervised learning

Abstract

Permutation of any two hidden units yields invariant properties in typical deep generative neural networks. This permutation symmetry plays an important role in understanding the computation performance of a broad class of neural networks with two or more hidden units. However, a theoretical study of the permutation symmetry is still lacking. Here, we propose a minimal model with only two hidden units in a restricted Boltzmann machine, which aims to address how the permutation symmetry affects the critical learning data size at which the concept-formation (or spontaneous symmetry breaking in physics language) starts, and moreover semi-rigorously prove a conjecture that the critical data size is independent of the number of hidden units once this number is finite. Remarkably, we find that the embedded correlation between two receptive fields of hidden units reduces the critical data size. In particular, the weakly-correlated receptive fields have the benefit of significantly reducing the minimal data size that triggers the transition, given less noisy data. Inspired by the theory, we also propose an efficient fully-distributed algorithm to infer the receptive fields of hidden units. Furthermore, our minimal model reveals that the permutation symmetry can also be spontaneously broken following the spontaneous symmetry breaking. Overall, our results demonstrate that the unsupervised learning is a progressive combination of spontaneous symmetry breaking and permutation symmetry breaking which are both spontaneous processes driven by data streams (observations). All these effects can be analytically probed based on the minimal model, providing theoretical insights towards understanding unsupervised learning in a more general context.

arXiv comment36 pages, 110 equations, 5 figures; a complete picture about physical laws of unsupervised learning in neural networks presented

References in corpus (13)

• The Bethe lattice spin glass revisited
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• High-Dimensional Inference with the generalized Hopfield Model: Principal Component Analysis and Corrections
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Nonequilibrium Thermodynamics of Restricted Boltzmann Machines
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Manifestation of Superposition and Coherence in $\mathcal{P}\mathcal{T}$-symmetry through the $\eta$-inner Product

Cited by in corpus (15)

• Statistical physics of unsupervised learning with prior knowledge in neural networks
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Eight challenges in developing theory of intelligence
• Weakly-correlated synapses promote dimension reduction in deep neural networks
• Coding schemes in neural networks learning classification tasks
• Nonequilbrium physics of generative diffusion models
• Dense Hopfield Networks in the Teacher-Student Setting
• Unsupervised Machine Learning of Quenched Gauge Symmetries: A Proof-of-Concept Demonstration
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• Ensemble perspective for understanding temporal credit assignment
• Associative memory model with arbitrary Hebbian length
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems
• Saddle Hierarchy in Dense Associative Memory