Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning

Abstract

The recently proposed Multi-Layer Convolutional Sparse Coding (ML-CSC) model, consisting of a cascade of convolutional sparse layers, provides a new interpretation of Convolutional Neural Networks (CNNs). Under this framework, the computation of the forward pass in a CNN is equivalent to a pursuit algorithm aiming to estimate the nested sparse representation vectors -- or feature maps -- from a given input signal. Despite having served as a pivotal connection between CNNs and sparse modeling, a deeper understanding of the ML-CSC is still lacking: there are no pursuit algorithms that can serve this model exactly, nor are there conditions to guarantee a non-empty model. While one can easily obtain signals that approximately satisfy the ML-CSC constraints, it remains unclear how to simply sample from the model and, more importantly, how one can train the convolutional filters from real data. In this work, we propose a sound pursuit algorithm for the ML-CSC model by adopting a projection approach. We provide new and improved bounds on the stability of the solution of such pursuit and we analyze different practical alternatives to implement this in practice. We show that the training of the filters is essential to allow for non-trivial signals in the model, and we derive an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers. Last, but not least, we demonstrate the applicability of the ML-CSC model for several applications in an unsupervised setting, providing competitive results. Our work represents a bridge between matrix factorization, sparse dictionary learning and sparse auto-encoders, and we analyze these connections in detail.

References in corpus (9)

• Winner-Take-All Autoencoders
• Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding
• Maximal Sparsity with Deep Networks?
• Trainlets: Dictionary Learning in High Dimensions
• A Probabilistic Theory of Deep Learning
• First and Second Order Methods for Online Convolutional Dictionary Learning
• Online Convolutional Dictionary Learning
• Convolutional Matching Pursuit and Dictionary Training
• A Probabilistic Framework for Deep Learning

Cited by in corpus (31)

• The Modern Mathematics of Deep Learning
• Transform Learning for Magnetic Resonance Image Reconstruction: From Model-based Learning to Building Neural Networks
• Algorithmic Guarantees for Inverse Imaging with Untrained Network Priors
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• First and Second Order Methods for Online Convolutional Dictionary Learning
• Unsupervised Deep Basis Pursuit: Learning inverse problems without ground-truth data
• Learning Deep Analysis Dictionaries for Image Super-Resolution
• A convex variational model for learning convolutional image atoms from incomplete data
• Learned convex regularizers for inverse problems
• Deep Networks from the Principle of Rate Reduction
• Interpretable Convolutional Neural Networks via Feedforward Design
• When and How Can Deep Generative Models be Inverted?
• Deep Convolutional Sparse Coding Networks for Image Fusion
• Deep Network Classification by Scattering and Homotopy Dictionary Learning
• Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations
• Deep Interpretable Non-Rigid Structure from Motion
• Multi-layer Residual Sparsifying Transform (MARS) Model for Low-dose CT Image Reconstruction
• Convolutional-Sparse-Coded Dynamic Mode Decomposition and Its Application to River State Estimation
• Pivotal Auto-Encoder via Self-Normalizing ReLU
• Deeply-Sparse Signal rePresentations ($\text{D}{\text{S}}^2\text{P}$)
• Learned Interpretable Residual Extragradient ISTA for Sparse Coding
• A Dictionary Approach to Domain-Invariant Learning in Deep Networks
• Deep learning based dictionary learning and tomographic image reconstruction
• Learning Deep Analysis Dictionaries -- Part II: Convolutional Dictionaries
• Convolutional Dictionary Learning in Hierarchical Networks
• A Sparse Coding Interpretation of Neural Networks and Theoretical Implications
• Sparsity-Probe: Analysis tool for Deep Learning Models
• Improved Image Generation via Sparse Modeling
• Two-layer Residual Sparsifying Transform Learning for Image Reconstruction
• From biological vision to unsupervised hierarchical sparse coding
• Sparse Coding Driven Deep Decision Tree Ensembles for Nuclear Segmentation in Digital Pathology Images