Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
arXiv:2410.16150 · doi:10.1016/j.neunet.2025.107542
Abstract
Restricted Boltzmann machines (RBM) are generative models capable to learn data with a rich underlying structure. We study the teacher-student setting where a student RBM learns structured data generated by a teacher RBM. The amount of structure in the data is controlled by adjusting the number of hidden units of the teacher and the correlations in the rows of the weights, a.k.a. patterns. In the absence of correlations, we validate the conjecture that the performance is independent of the number of teacher patters and hidden units of the student RBMs, and we argue that the teacher-student setting can be used as a toy model for studying the lottery ticket hypothesis. Beyond this regime, we find that the critical amount of data required to learn the teacher patterns decreases with both their number and correlations. In both regimes, we find that, even with a relatively large dataset, it becomes impossible to learn the teacher patterns if the inference temperature used for regularization is kept too low. In our framework, the student can learn teacher patterns one-to-one or many-to-one, generalizing previous findings about the teacher-student setting with two hidden units to any arbitrary finite number of hidden units.
24 pages, 21 figures
References in corpus (30)
- Optimal whitening and decorrelation
- Emergence of Compositional Representations in Restricted Boltzmann Machines
- Modelling the influence of data structure on learning in neural networks: the hidden manifold model
- The free energy in a multi-species Sherrington-Kirkpatrick model
- Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence (2013)
- Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
- Extensive load in multitasking associative networks
- Phase transitions in Restricted Boltzmann Machines with generic priors
- Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
- From Dyson to Hopfield: Processing on hierarchical networks
- Spectral Dynamics of Learning Restricted Boltzmann Machines
- How glassy are neural networks?
- Non-Convex Multi-species Hopfield models
- Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
- Hierarchical neural networks perform both serial and parallel processing
- Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
- Minimal model of permutation symmetry in unsupervised learning
- On the Optimal Weighted Regularization in Overparameterized Linear Regression
- Phase diagram for two-layer ReLU neural networks at infinite-width limit
- Hopfield model with planted patterns: a teacher-student self-supervised learning model
- Legendre Equivalences of Spherical Boltzmann Machines
- Non-Convex Multipartite Ferromagnets
- Dense Hopfield Networks in the Teacher-Student Setting
- Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
- Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
- Role of zero synapses in unsupervised feature learning
- On the interplay between data structure and loss function in classification problems
- The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
- Overlap synchronisation in multipartite random energy models
- From the replica trick to the replica symmetry breaking technique