Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures

Abstract

Model-based methods and deep neural networks have both been tremendously successful paradigms in machine learning. In model-based methods, problem domain knowledge can be built into the constraints of the model, typically at the expense of difficulties during inference. In contrast, deterministic deep neural networks are constructed in such a way that inference is straightforward, but their architectures are generic and it is unclear how to incorporate knowledge. This work aims to obtain the advantages of both approaches. To do so, we start with a model-based approach and an associated inference algorithm, and \emph{unfold} the inference iterations as layers in a deep network. Rather than optimizing the original model, we \emph{untie} the model parameters across layers, in order to create a more powerful network. The resulting architecture can be trained discriminatively to perform accurate inference within a fixed network size. We show how this framework allows us to interpret conventional networks as mean-field inference in Markov random fields, and to obtain new architectures by instead using belief propagation as the inference algorithm. We then show its application to a non-negative matrix factorization model that incorporates the problem-domain knowledge that sound sources are additive. Deep unfolding of this model yields a new kind of non-negative deep neural network, that can be trained using a multiplicative backpropagation-style update algorithm. We present speech enhancement experiments showing that our approach is competitive with conventional neural networks despite using far fewer parameters.

Added sections on reducing belief propagation to network activation functions, and on conversion between conventional network parameters and BP potentials for binary MRFs. Some bugs and typos were also fixed, and notation made a bit clearer

References in corpus (2)

• Variational Inference in Non-negative Factorial Hidden Markov Models for Efficient Audio Source Separation
• Negative Tree Reweighted Belief Propagation

Cited by in corpus (27)

• On the Explainability of Natural Language Processing Deep Models
• ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI
• GAP-net for Snapshot Compressive Imaging
• Neural Expectation Maximization
• Deep Learning-Guided Image Reconstruction from Incomplete Data
• Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery
• Model-Aware Deep Architectures for One-Bit Compressive Variational Autoencoding
• Deep unfolding of the weighted MMSE beamforming algorithm
• Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descent
• Unfolded Algorithms for Deep Phase Retrieval
• Learning for Integer-Constrained Optimization through Neural Networks with Limited Training
• Learning to Beamform in Heterogeneous Massive MIMO Networks
• Deep-RLS: A Model-Inspired Deep Learning Approach to Nonlinear PCA
• Learning step sizes for unfolded sparse coding
• Deep Griffin-Lim Iteration
• Compressive-Sensing Data Reconstruction for Structural Health Monitoring: A Machine-Learning Approach
• ADMM-based Decoder for Binary Linear Codes Aided by Deep Learning
• Closing the Training/Inference Gap for Deep Attractor Networks
• Resource Allocation Using Gradient Boosting Aided Deep Q-Network for IoT in C-RANs
• Sparse Array Beamformer Design for Active and Passive Sensing
• Deep Unfolded Multicast Beamforming
• Learned Interpretable Residual Extragradient ISTA for Sparse Coding
• A Low-Complexity MIMO Channel Estimator with Implicit Structure of a Convolutional Neural Network
• Interpretable Deep Multimodal Image Super-Resolution
• Sparse Array Capon Beamformer Design Availing Deep Learning
• Tree Search Network for Sparse Regression
• A Low-Complexity ADMM-based Massive MIMO Detectors via Deep Neural Networks