Tensorial Mixture Models

Abstract

Casting neural networks in generative frameworks is a highly sought-after endeavor these days. Contemporary methods, such as Generative Adversarial Networks, capture some of the generative capabilities, but not all. In particular, they lack the ability of tractable marginalization, and thus are not suitable for many tasks. Other methods, based on arithmetic circuits and sum-product networks, do allow tractable marginalization, but their performance is challenged by the need to learn the structure of a circuit. Building on the tractability of arithmetic circuits, we leverage concepts from tensor analysis, and derive a family of generative models we call Tensorial Mixture Models (TMMs). TMMs assume a simple convolutional network structure, and in addition, lend themselves to theoretical analyses that allow comprehensive understanding of the relation between their structure and their expressive properties. We thus obtain a generative model that is tractable on one hand, and on the other hand, allows effective representation of rich distributions in an easily controlled manner. These two capabilities are brought together in the task of classification under missing data, where TMMs deliver state of the art accuracies with seamless implementation and design.

arXiv commentA git repository for reproducing our experiments is available at: https://github.com/HUJI-Deep/Generative-ConvACs

References in corpus (11)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Probabilistic Latent Semantic Analysis
• NICE: Non-linear Independent Components Estimation
• Improved Techniques for Training GANs
• Sum-Product Networks: A New Deep Architecture
• Deep Generative Stochastic Networks Trainable by Backprop
• Improving Variational Inference with Inverse Autoregressive Flow
• A Survey on Latent Tree Models and Applications
• SimNets: A Generalization of Convolutional Networks

Cited by in corpus (9)

• Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Inductive Bias of Deep Convolutional Networks through Pooling Geometry
• Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design
• Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions
• Bayesian Learning of Sum-Product Networks
• TGAN: Deep Tensor Generative Adversarial Nets for Large Image Generation
• Deep Generalized Convolutional Sum-Product Networks
• Implicit Regularization in Tensor Factorization