A Probabilistic Framework for Deep Learning

Abstract

We develop a probabilistic framework for deep learning based on the Deep Rendering Mixture Model (DRMM), a new generative probabilistic model that explicitly capture variations in data due to latent task nuisance variables. We demonstrate that max-sum inference in the DRMM yields an algorithm that exactly reproduces the operations in deep convolutional neural networks (DCNs), providing a first principles derivation. Our framework provides new insights into the successes and shortcomings of DCNs as well as a principled route to their improvement. DRMM training via the Expectation-Maximization (EM) algorithm is a powerful alternative to DCN back-propagation, and initial training results are promising. Classification based on the DRMM and other variants outperforms DCNs in supervised digit classification, training 2-3x faster while achieving similar accuracy. Moreover, the DRMM is applicable to semi-supervised and unsupervised learning tasks, achieving results that are state-of-the-art in several categories on the MNIST benchmark and comparable to state of the art on the CIFAR10 benchmark.

arXiv admin note: substantial text overlap with arXiv:1504.00641

References in corpus (11)

• Auto-Encoding Variational Bayes
• Semi-Supervised Learning with Deep Generative Models
• Improved Techniques for Training GANs
• On the Number of Linear Regions of Deep Neural Networks
• Semi-Supervised Learning with Ladder Networks
• Auxiliary Deep Generative Models
• Provable Bounds for Learning Some Deep Representations
• Winner-Take-All Autoencoders
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• A Probabilistic Theory of Deep Learning
• Deep Mixtures of Factor Analysers

Cited by in corpus (7)

• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• The Gaussian equivalence of generative models for learning with shallow neural networks
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• A synthetic dataset for deep learning
• A Probabilistic Representation of Deep Learning
• Explicitly Bayesian Regularizations in Deep Learning
• Rediscovering Deep Neural Networks Through Finite-State Distributions