Expressive Power and Approximation Errors of Restricted Boltzmann Machines

Abstract

We present explicit classes of probability distributions that can be learned by Restricted Boltzmann Machines (RBMs) depending on the number of units that they contain, and which are representative for the expressive power of the model. We use this to show that the maximal Kullback-Leibler divergence to the RBM model with $n$ visible and $m$ hidden units is bounded from above by $n-\lfloor \log (m+1)\rfloor -\frac{m+1}{2^{\lfloor \log (m+1)\rfloor }}\approx (n-1)-\log (m+1)$. In this way we can specify the number of hidden units that guarantees a sufficiently rich model containing different classes of distributions and respecting a given error tolerance.

arXiv comment9 pages, 3 figures, plus 1 page, 1 figure appendix, minor corrections of the first publication

References in corpus (2)

• Maximizing Multi-Information
• Support Sets in Exponential Families and Oriented Matroid Theory

Cited by in corpus (15)

• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
• Machine learning with neural networks
• Probabilistic Modeling with Matrix Product States
• A Theory of Cheap Control in Embodied Systems
• Natural evolution strategies and variational Monte Carlo
• Symmetric Tensor Networks for Generative Modeling and Constrained Combinatorial Optimization
• Geometry and Expressive Power of Conditional Restricted Boltzmann Machines
• Representational aspects of depth and conditioning in normalizing flows
• Scaling of Model Approximation Errors and Expected Entropy Distances
• Discrete Restricted Boltzmann Machines
• Mixture decompositions of exponential families using a decomposition of their sample spaces
• Kernels and Submodels of Deep Belief Networks
• Maximizing the Bregman divergence from a Bregman family
• Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units