Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function

Abstract

We study the expectation value of the logarithm of the partition function of large binary-to-binary lattice-gas Restricted Boltzmann Machines (RBMs) within a replica-symmetric ansatz, averaging over the disorder represented by the parameters of the RBM Hamiltonian. Averaging over the Hamiltonian parameters is done with a diagonal covariance matrix. Due to the diagonal form of the parameter covariance matrix not being preserved under the isomorphism between the Ising and lattice-gas forms of the RBM, we find differences in the behaviour of the quenched log partition function of the lattice-gas RBM compared to that of the Ising RBM form usually studied. We obtain explicit expressions for the expectation and variance of the lattice-gas RBM log partition function per node in the thermodynamic limit. We also obtain explicit expressions for the leading order finite size correction to the expected log partition function per node, and the threshold for the stability of the replica-symmetric approximation. We show that the stability threshold of the replica-symmetric approximation is equivalent, in the thermodynamic limit, to the stability threshold of a recent message-passing algorithm used to construct a mean-field Bethe approximation to the RBM free energy. Given the replica-symmetry assumption breaks down as the level of disorder in the spin-spin couplings increases, we obtain asymptotic expansions, in terms of the variance controlling this disorder, for the replica-symmetric log partition function and the replica-symmetric stability threshold. We confirm the various results derived using simulation.

arXiv comment28 pages, 6 figures; Reference Zhao et al, 2022, added; In response to referees comments - 2 extra figures added, parts of section VI re-worded, discussion extended

References in corpus (14)

• Glass models on Bethe lattices
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Equilibrium statistical mechanics of bipartite spin systems
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Propagating beliefs in spin glass models
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• A Friendly Tutorial on Mean-Field Spin Glass Techniques for Non-Physicists
• Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems
• Lattice Gas Analogue Of SK Model: A paradigm for the glass transition