On the role of non-linear latent features in bipartite generative neural networks

Abstract

We investigate the phase diagram and memory retrieval capabilities of bipartite energy-based neural networks, namely Restricted Boltzmann Machines (RBMs), as a function of the prior distribution imposed on their hidden units - including binary, multi-state, and ReLU-like activations. Drawing connections to the Hopfield model and employing analytical tools from statistical physics of disordered systems, we explore how the architectural choices and activation functions shape the thermodynamic properties of these models. Our analysis reveals that standard RBMs with binary hidden nodes and extensive connectivity suffer from reduced critical capacity, limiting their effectiveness as associative memories. To address this, we examine several modifications, such as introducing local biases and adopting richer hidden unit priors. These adjustments restore ordered retrieval phases and markedly improve recall performance, even at finite temperatures. Our theoretical findings, supported by finite-size Monte Carlo simulations, highlight the importance of hidden unit design in enhancing the expressive power of RBMs.

arXiv comment23 pages, 5 figures

References in corpus (31)

• The physics of higher-order interactions in complex systems
• Inverse Statistical Physics of Protein Sequences: A Key Issues Review
• Pairwise maximum entropy models for studying large biological systems: when they can and when they can't work
• Inverse Ising inference using all the data
• Statistical physics of pairwise probability models
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Mean-field message-passing equations in the Hopfield model and its generalizations
• The Bethe approximation for solving the inverse Ising problem: a comparison with other inference methods
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Pseudolikelihood Decimation Algorithm Improving the Inference of the Interaction Network in a General Class of Ising Models
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Mean-field theory for the inverse Ising problem at low temperatures
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Replica Symmetry Breaking in Attractor Neural Network Models
• Machine learning determination of dynamical parameters: The Ising model case
• Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
• Dreaming neural networks: rigorous results
• Replica symmetry breaking in dense neural networks
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Solving the inverse Ising problem by mean-field methods in a clustered phase space with many states
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Inferring effective couplings with Restricted Boltzmann Machines
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• The effect of priors on Learning with Restricted Boltzmann Machines
• A fast and accurate algorithm for inferring sparse Ising models via parameters activation to maximize the pseudo-likelihood
• Inferring Higher-Order Couplings with Neural Networks