A Probabilistic Theory of Deep Learning

Abstract

A grand challenge in machine learning is the development of computational algorithms that match or outperform humans in perceptual inference tasks that are complicated by nuisance variation. For instance, visual object recognition involves the unknown object position, orientation, and scale in object recognition while speech recognition involves the unknown voice pronunciation, pitch, and speed. Recently, a new breed of deep learning algorithms have emerged for high-nuisance inference tasks that routinely yield pattern recognition systems with near- or super-human capabilities. But a fundamental question remains: Why do they work? Intuitions abound, but a coherent framework for understanding, analyzing, and synthesizing deep learning architectures has remained elusive. We answer this question by developing a new probabilistic framework for deep learning based on the Deep Rendering Model: a generative probabilistic model that explicitly captures latent nuisance variation. By relaxing the generative model to a discriminative one, we can recover two of the current leading deep learning systems, deep convolutional neural networks and random decision forests, providing insights into their successes and shortcomings, as well as a principled route to their improvement.

56 pages, 6 figures, 2 tables

References in corpus (9)

• Deep Learning in Neural Networks: An Overview
• Improving neural networks by preventing co-adaptation of feature detectors
• Going Deeper with Convolutions
• Online EM Algorithm for Latent Data Models
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Unsupervised Learning of Invariant Representations in Hierarchical Architectures
• An Unsupervised Algorithm For Learning Lie Group Transformations
• On Invariance and Selectivity in Representation Learning
• Modeling sequential data using higher-order relational features and predictive training

Cited by in corpus (23)

• Provable approximation properties for deep neural networks
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• Deep Learning for Passive Synthetic Aperture Radar
• Visual Representations: Defining Properties and Deep Approximations
• A Probabilistic Framework for Deep Learning
• When are Deep Networks really better than Decision Forests at small sample sizes, and how?
• Truncated Variational Expectation Maximization
• Deep Learning and Hierarchal Generative Models
• SLiMFast: Guaranteed Results for Data Fusion and Source Reliability
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• Distributed Sequence Memory of Multidimensional Inputs in Recurrent Networks
• On architectural choices in deep learning: From network structure to gradient convergence and parameter estimation
• Semi-Supervised Learning with the Deep Rendering Mixture Model
• Between Homomorphic Signal Processing and Deep Neural Networks: Constructing Deep Algorithms for Polyphonic Music Transcription
• Fractal Dimension Invariant Filtering and Its CNN-based Implementation
• A Simple Hierarchical Pooling Data Structure for Loop Closure
• Semi-Supervised Learning via New Deep Network Inversion
• Latent Regression Bayesian Network for Data Representation
• Deeply-Sparse Signal rePresentations ($\text{D}{\text{S}}^2\text{P}$)
• The staircase property: How hierarchical structure can guide deep learning
• Face representation by deep learning: a linear encoding in a parameter space?
• PCANet: An energy perspective