Statistical physics of unsupervised learning with prior knowledge in neural networks

Abstract

Integrating sensory inputs with prior beliefs from past experiences in unsupervised learning is a common and fundamental characteristic of brain or artificial neural computation. However, a quantitative role of prior knowledge in unsupervised learning remains unclear, prohibiting a scientific understanding of unsupervised learning. Here, we propose a statistical physics model of unsupervised learning with prior knowledge, revealing that the sensory inputs drive a series of continuous phase transitions related to spontaneous intrinsic-symmetry breaking. The intrinsic symmetry includes both reverse symmetry and permutation symmetry, commonly observed in most artificial neural networks. Compared to the prior-free scenario, the prior reduces more strongly the minimal data size triggering the reverse symmetry breaking transition, and moreover, the prior merges, rather than separates, permutation symmetry breaking phases. We claim that the prior can be learned from data samples, which in physics corresponds to a two-parameter Nishimori constraint. This work thus reveals mechanisms about the influence of the prior on unsupervised learning.

arXiv comment15 pages, 3 figures with supplemental material

References in corpus (6)

• The Bethe lattice spin glass revisited
• The Nishimori line and Bayesian Statistics
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Minimal model of permutation symmetry in unsupervised learning

Cited by in corpus (8)

• Eight challenges in developing theory of intelligence
• Nonequilbrium physics of generative diffusion models
• Weakly-correlated synapses promote dimension reduction in deep neural networks
• Associative memory model with arbitrary Hebbian length
• Ensemble perspective for understanding temporal credit assignment
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Analyticity of the energy in an Ising spin glass with correlated disorder
• Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems