Deep Residual Autoencoders for Expectation Maximization-inspired Dictionary Learning

Abstract

We introduce a neural-network architecture, termed the constrained recurrent sparse autoencoder (CRsAE), that solves convolutional dictionary learning problems, thus establishing a link between dictionary learning and neural networks. Specifically, we leverage the interpretation of the alternating-minimization algorithm for dictionary learning as an approximate Expectation-Maximization algorithm to develop autoencoders that enable the simultaneous training of the dictionary and regularization parameter (ReLU bias). The forward pass of the encoder approximates the sufficient statistics of the E-step as the solution to a sparse coding problem, using an iterative proximal gradient algorithm called FISTA. The encoder can be interpreted either as a recurrent neural network or as a deep residual network, with two-sided ReLU non-linearities in both cases. The M-step is implemented via a two-stage back-propagation. The first stage relies on a linear decoder applied to the encoder and a norm-squared loss. It parallels the dictionary update step in dictionary learning. The second stage updates the regularization parameter by applying a loss function to the encoder that includes a prior on the parameter motivated by Bayesian statistics. We demonstrate in an image-denoising task that CRsAE learns Gabor-like filters, and that the EM-inspired approach for learning biases is superior to the conventional approach. In an application to recordings of electrical activity from the brain, we demonstrate that CRsAE learns realistic spike templates and speeds up the process of identifying spike times by 900x compared to algorithms based on convex optimization.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• On the Convergence of Adam and Beyond
• Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures
• Convolutional Dictionary Learning: A Comparative Review and New Algorithms
• Learned Convolutional Sparse Coding
• Neural Proximal Gradient Descent for Compressive Imaging
• Rethinking the CSC Model for Natural Images
• Neural Network Alternatives to Convolutive Audio Models for Source Separation

Cited by in corpus (11)

• Extended Successive Convex Approximation for Phase Retrieval with Dictionary Learning
• Unrolled Compressed Blind-Deconvolution
• Adversarial Robustness of Supervised Sparse Coding
• DuRIN: A Deep-unfolded Sparse Seismic Reflectivity Inversion Network
• Deeply-Sparse Signal rePresentations ($\text{D}{\text{S}}^2\text{P}$)
• Discriminative reconstruction via simultaneous dense and sparse coding
• NuSPAN: A Proximal Average Network for Nonuniform Sparse Model -- Application to Seismic Reflectivity Inversion
• Deep learning based dictionary learning and tomographic image reconstruction
• Convolutional dictionary learning based auto-encoders for natural exponential-family distributions
• Multi-local Collaborative AutoEncoder
• RandNet: deep learning with compressed measurements of images