Inference in Multi-Layer Networks with Matrix-Valued Unknowns

Abstract

We consider the problem of inferring the input and hidden variables of a stochastic multi-layer neural network from an observation of the output. The hidden variables in each layer are represented as matrices. This problem applies to signal recovery via deep generative prior models, multi-task and mixed regression and learning certain classes of two-layer neural networks. A unified approximation algorithm for both MAP and MMSE inference is proposed by extending a recently-developed Multi-Layer Vector Approximate Message Passing (ML-VAMP) algorithm to handle matrix-valued unknowns. It is shown that the performance of the proposed Multi-Layer Matrix VAMP (ML-Mat-VAMP) algorithm can be exactly predicted in a certain random large-system limit, where the dimensions $N\times d$ of the unknown quantities grow as $N\to \infty$ with $d$ fixed. In the two-layer neural-network learning problem, this scaling corresponds to the case where the number of input features and training samples grow to infinity but the number of hidden nodes stays fixed. The analysis enables a precise prediction of the parameter and test error of the learning.

arXiv comment3 figures, 6 pages (two-column) + Appendix. arXiv admin note: text overlap with arXiv:1911.03409

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Message Passing Algorithms for Compressed Sensing
• Expectation Propagation for approximate Bayesian inference
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Compressed Sensing using Generative Models
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Efficient High-Dimensional Inference in the Multiple Measurement Vector Problem
• Compressed Sensing with Deep Image Prior and Learned Regularization
• Entropy and mutual information in models of deep neural networks
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Alternating Minimization for Mixed Linear Regression
• Correction by Projection: Denoising Images with Generative Adversarial Networks
• Task-Aware Compressed Sensing with Generative Adversarial Networks
• SUNLayer: Stable denoising with generative networks
• Approximate message-passing for convex optimization with non-separable penalties

Cited by in corpus (1)

• Multi-Layer Bilinear Generalized Approximate Message Passing