Entropy, Free Energy, and Work of Restricted Boltzmann Machines
arXiv:2004.04900 · doi:10.3390/e22050538
Abstract
A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing parameters of the energy function of the network. In this paper, we analyze the training process of the restricted Boltzmann machine in the context of statistical physics. As an illustration, for small size Bar-and-Stripe patterns, we calculate thermodynamic quantities such as entropy, free energy, and internal energy as a function of training epoch. We demonstrate the growth of the correlation between the visible and hidden layers via the subadditivity of entropies as the training proceeds. Using the Monte-Carlo simulation of trajectories of the visible and hidden vectors in configuration space, we also calculate the distribution of the work done on the restricted Boltzmann machine by switching the parameters of the energy function. We discuss the Jarzynski equality which connects the path average of the exponential function of the work and the difference in free energies before and after training.
arXiv comment10 figures, submitted
References in corpus (17)
- A nonequilibrium equality for free energy differences
- The Entropy Production Fluctuation Theorem and the Nonequilibrium Work Relation for Free Energy Differences
- Solving the Quantum Many-Body Problem with Artificial Neural Networks
- Path ensembles averages in systems driven far-from-equilibrium
- A high-bias, low-variance introduction to Machine Learning for physicists
- Efficient Representation of Quantum Many-body States with Deep Neural Networks
- Rare events and the convergence of exponentially averaged work values
- Towards Quantum Machine Learning with Tensor Networks
- Equivalence of restricted Boltzmann machines and tensor network states
- Theory of Systematic Computational Error in Free Energy Differences
- A "fast growth" method of computing free energy differences
- Machine learning meets quantum physics
- Quantum Machine Learning for Electronic Structure Calculations
- Optimized free energies from bidirectional single-molecule force spectroscopy
- Equilibrium free energies from fast-switching trajectories with large time steps
- On the efficiency of path sampling methods for the calculation of free energies from non-equilibrium simulations
- Number of trials required to estimate a free-energy difference, using fluctuation relations