Subtractive Mixture Models via Squaring: Representation and Learning

Abstract

Mixture models are traditionally represented and learned by adding several distributions as components. Allowing mixtures to subtract probability mass or density can drastically reduce the number of components needed to model complex distributions. However, learning such subtractive mixtures while ensuring they still encode a non-negative function is challenging. We investigate how to learn and perform inference on deep subtractive mixtures by squaring them. We do this in the framework of probabilistic circuits, which enable us to represent tensorized mixtures and generalize several other subtractive models. We theoretically prove that the class of squared circuits allowing subtractions can be exponentially more expressive than traditional additive mixtures; and, we empirically show this increased expressiveness on a series of real-world distribution estimation tasks.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Boosted Decision Trees as an Alternative to Artificial Neural Networks for Particle Identification
• Differentiable Learning of Quantum Circuit Born Machine
• Unsupervised Generative Modeling Using Matrix Product States
• Tree Tensor Networks for Generative Modeling
• Approximation by finite mixtures of continuous density functions that vanish at infinity
• On the Expressive Efficiency of Sum Product Networks
• Semantic Probabilistic Layers for Neuro-Symbolic Learning
• Non-parametric Models for Non-negative Functions
• Tractable Regularization of Probabilistic Circuits
• Scaling Up Probabilistic Circuits by Latent Variable Distillation
• Learning Negative Mixture Models by Tensor Decompositions
• Probabilistic Circuits for Variational Inference in Discrete Graphical Models
• Squared Neural Families: A New Class of Tractable Density Models
• Polynomial Semantics of Tractable Probabilistic Circuits