The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications

Abstract

Restricted Boltzmann Machines are simple yet powerful neural networks. They can be used for learning structure in data, and are used as a building block of more complex neural architectures. At the same time, their simplicity makes them easy to use, amenable to theoretical analysis, yielding interpretable models in applications. Here, we focus on reviewing the role that the activation functions, describing the input-output relationship of single neurons in RBM, play in the functionality of these models. We discuss recent theoretical results on the benefits and limitations of different activation functions. We also review applications to biological data analysis, namely neural data analysis, where RBM units are mostly taken to have sigmoid activation functions and binary units, to protein data analysis and immunology where non-binary units and non-sigmoid activation functions have recently been shown to yield important insights into the data. Finally, we discuss open problems addressing which can shed light on broader issues in neural network research.

arXiv comment7 pages, 3 figures. To be published in EPL as di Sarra et al 2025 EPL. Accepted manuscript available online at https://doi.org/10.1209/0295-5075/ada636

References in corpus (17)

• Solving the Quantum Many-Body Problem with Artificial Neural Networks
• Direct-coupling analysis of residue co-evolution captures native contacts across many protein families
• Neural-Network Approach to Dissipative Quantum Many-Body Dynamics
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Supervised Hebbian Learning
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Selection of sequence motifs and generative Hopfield-Potts models for protein familiesilies
• Machine learning determination of dynamical parameters: The Ising model case
• The Exponential Capacity of Dense Associative Memories
• Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
• Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
• Unsupervised hierarchical clustering using the learning dynamics of RBMs
• Blindfold learning of an accurate neural metric
• Inferring effective couplings with Restricted Boltzmann Machines
• Efficiency of local learning rules in threshold-linear associative networks

Cited by in corpus (3)

• Inferring Higher-Order Couplings with Neural Networks
• On the role of non-linear latent features in bipartite generative neural networks
• Supervised and Unsupervised protocols for hetero-associative neural networks