Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition

Abstract

Unsupervised neural network learning extracts hidden features from unlabeled training data. This is used as a pretraining step for further supervised learning in deep networks. Hence, understanding unsupervised learning is of fundamental importance. Here, we study the unsupervised learning from a finite number of data, based on the restricted Boltzmann machine learning. Our study inspires an efficient message passing algorithm to infer the hidden feature, and estimate the entropy of candidate features consistent with the data. Our analysis reveals that the learning requires only a few data if the feature is salient and extensively many if the feature is weak. Moreover, the entropy of candidate features monotonically decreases with data size and becomes negative (i.e., entropy crisis) before the message passing becomes unstable, suggesting a discontinuous phase transition. In terms of convergence time of the message passing algorithm, the unsupervised learning exhibits an easy-hard-easy phenomenon as the training data size increases. All these properties are reproduced in an approximate Hopfield model, with an exception that the entropy crisis is absent, and only continuous phase transition is observed. This key difference is also confirmed in a handwritten digits dataset. This study deepens our understanding of unsupervised learning from a finite number of data, and may provide insights into its role in training deep networks.

8 pages, 7 figures (5 pages, 4 figures in the main text and 3 pages of appendix)

References in corpus (10)

• The Bethe lattice spin glass revisited
• Learning by message-passing in networks of discrete synapses
• Multitasking associative networks
• The Nishimori line and Bayesian Statistics
• The glassy phase of Gallager codes
• Origin of the computational hardness for learning with binary synapses
• High-Dimensional Inference with the generalized Hopfield Model: Principal Component Analysis and Corrections
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• A frozen glass phase in the multi-index matching problem
• Code optimization, frozen glassy phase and improved decoding algorithms for low-density parity-check codes

Cited by in corpus (17)

• Restricted Boltzmann Machine, recent advances and mean-field theory
• Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Nishimori meets Bethe: a spectral method for node classification in sparse weighted graphs
• Statistical physics of unsupervised learning with prior knowledge in neural networks
• Minimal model of permutation symmetry in unsupervised learning
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Role of zero synapses in unsupervised feature learning
• The effect of priors on Learning with Restricted Boltzmann Machines
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Analyticity of the energy in an Ising spin glass with correlated disorder
• On the accuracy of message-passing approaches to percolation in complex networks
• Saddle Hierarchy in Dense Associative Memory