Bayesian Learning of Sum-Product Networks

Abstract

Sum-product networks (SPNs) are flexible density estimators and have received significant attention due to their attractive inference properties. While parameter learning in SPNs is well developed, structure learning leaves something to be desired: Even though there is a plethora of SPN structure learners, most of them are somewhat ad-hoc and based on intuition rather than a clear learning principle. In this paper, we introduce a well-principled Bayesian framework for SPN structure learning. First, we decompose the problem into i) laying out a computational graph, and ii) learning the so-called scope function over the graph. The first is rather unproblematic and akin to neural network architecture validation. The second represents the effective structure of the SPN and needs to respect the usual structural constraints in SPN, i.e. completeness and decomposability. While representing and learning the scope function is somewhat involved in general, in this paper, we propose a natural parametrisation for an important and widely used special case of SPNs. These structural parameters are incorporated into a Bayesian model, such that simultaneous structure and parameter learning is cast into monolithic Bayesian posterior inference. In various experiments, our Bayesian SPNs often improve test likelihoods over greedy SPN learners. Further, since the Bayesian framework protects against overfitting, we can evaluate hyper-parameters directly on the Bayesian model score, waiving the need for a separate validation set, which is especially beneficial in low data regimes. Bayesian SPNs can be applied to heterogeneous domains and can easily be extended to nonparametric formulations. Moreover, our Bayesian approach is the first, which consistently and robustly learns SPN structures under missing data.

arXiv commentNeurIPS 2019; See conference page for supplement

References in corpus (7)

• Neural Architecture Search with Reinforcement Learning
• Sum-Product Networks: A New Deep Architecture
• SPFlow: An Easy and Extensible Library for Deep Probabilistic Learning using Sum-Product Networks
• A Unified Approach for Learning the Parameters of Sum-Product Networks
• Tensorial Mixture Models
• Optimisation of Overparametrized Sum-Product Networks
• Maximum margin learning of t-SPNs for cell classification with filtered input

Cited by in corpus (6)

• SPPL: Probabilistic Programming with Fast Exact Symbolic Inference
• Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits
• Deep Structured Mixtures of Gaussian Processes
• Sum-product networks: A survey
• Recursive Bayesian Networks: Generalising and Unifying Probabilistic Context-Free Grammars and Dynamic Bayesian Networks
• On the Sample Complexity of Learning Sum-Product Networks