A topological insight into restricted Boltzmann machines

Abstract

Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

http://link.springer.com/article/10.1007/s10994-016-5570-z, Machine Learning, issn=1573-0565, 2016

References in corpus (6)

• Power-law distributions in empirical data
• Do Deep Nets Really Need to be Deep?
• Understanding brain networks and brain organization
• MADE: Masked Autoencoder for Distribution Estimation
• A topological insight into restricted Boltzmann machines
• Sparse Group Restricted Boltzmann Machines

Cited by in corpus (16)

• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Equivalence of restricted Boltzmann machines and tensor network states
• Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• A topological insight into restricted Boltzmann machines
• SpaceNet: Make Free Space For Continual Learning
• The learnability scaling of quantum states: restricted Boltzmann machines
• Simulating disordered quantum systems via dense and sparse restricted Boltzmann machines
• Estimating 3D Trajectories from 2D Projections via Disjunctive Factored Four-Way Conditional Restricted Boltzmann Machines
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Selfish Sparse RNN Training
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware
• Topological Insights into Sparse Neural Networks
• Self-building Neural Networks