Inference in Deep Networks in High Dimensions

Abstract

Deep generative networks provide a powerful tool for modeling complex data in a wide range of applications. In inverse problems that use these networks as generative priors on data, one must often perform inference of the inputs of the networks from the outputs. Inference is also required for sampling during stochastic training on these generative models. This paper considers inference in a deep stochastic neural network where the parameters (e.g., weights, biases and activation functions) are known and the problem is to estimate the values of the input and hidden units from the output. While several approximate algorithms have been proposed for this task, there are few analytic tools that can provide rigorous guarantees in the reconstruction error. This work presents a novel and computationally tractable output-to-input inference method called Multi-Layer Vector Approximate Message Passing (ML-VAMP). The proposed algorithm, derived from expectation propagation, extends earlier AMP methods that are known to achieve the replica predictions for optimality in simple linear inverse problems. Our main contribution shows that the mean-squared error (MSE) of ML-VAMP can be exactly predicted in a certain large system limit (LSL) where the numbers of layers is fixed and weight matrices are random and orthogonally-invariant with dimensions that grow to infinity. ML-VAMP is thus a principled method for output-to-input inference in deep networks with a rigorous and precise performance achievability result in high dimensions.

27 pages

References in corpus (16)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Message Passing Algorithms for Compressed Sensing
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Loss Surfaces of Multilayer Networks
• Adversarially Learned Inference
• Spectral redemption: clustering sparse networks
• Expectation-Maximization Gaussian-Mixture Approximate Message Passing
• Deep Generative Stochastic Networks Trainable by Backprop
• Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing
• Adaptive Damping and Mean Removal for the Generalized Approximate Message Passing Algorithm
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Hybrid Approximate Message Passing
• Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines
• Expectation Propagation for Neural Networks with Sparsity-promoting Priors
• Generalized Expectation Consistent Signal Recovery for Nonlinear Measurements

Cited by in corpus (8)

• Compressed Sensing with Deep Image Prior and Learned Regularization
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Entropy and mutual information in models of deep neural networks
• Inverting Deep Generative models, One layer at a time
• Estimation for High-Dimensional Multi-Layer Generalized Linear Model -- Part I: The Exact MMSE Estimator
• Symbol Detection for Massive MIMO AF Relays Using Approximate Bayesian Inference
• Towards Designing Optimal Sensing Matrices for Generalized Linear Inverse Problems
• Gridless Variational Bayesian Channel Estimation for Antenna Array Systems with Low Resolution ADCs