Learning the Structure of Deep Sparse Graphical Models

Abstract

Deep belief networks are a powerful way to model complex probability distributions. However, learning the structure of a belief network, particularly one with hidden units, is difficult. The Indian buffet process has been used as a nonparametric Bayesian prior on the directed structure of a belief network with a single infinitely wide hidden layer. In this paper, we introduce the cascading Indian buffet process (CIBP), which provides a nonparametric prior on the structure of a layered, directed belief network that is unbounded in both depth and width, yet allows tractable inference. We use the CIBP prior with the nonlinear Gaussian belief network so each unit can additionally vary its behavior between discrete and continuous representations. We provide Markov chain Monte Carlo algorithms for inference in these belief networks and explore the structures learned on several image data sets.

20 pages, 6 figures, AISTATS 2010, Revised

References in corpus (1)

• The Bayesian Structural EM Algorithm

Cited by in corpus (20)

• Sum-Product Networks: A New Deep Architecture
• Avoiding pathologies in very deep networks
• Discovering Structure in High-Dimensional Data Through Correlation Explanation
• MaskConnect: Connectivity Learning by Gradient Descent
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Connectivity Learning in Multi-Branch Networks
• Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow
• Learning to Generate with Memory
• Constructing Deep Neural Networks by Bayesian Network Structure Learning
• Big Learning with Bayesian Methods
• Learning Deep Generative Models with Doubly Stochastic MCMC
• Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement
• Scalable multiscale density estimation
• Learning Sparse Deep Feedforward Networks via Tree Skeleton Expansion
• Beta diffusion trees and hierarchical feature allocations
• The Indian Chefs Process
• Building Sparse Deep Feedforward Networks using Tree Receptive Fields
• Non-parametric Bayesian Learning with Deep Learning Structure and Its Applications in Wireless Networks