Convolutional Neural Networks Analyzed via Convolutional Sparse Coding

Abstract

Convolutional neural networks (CNN) have led to many state-of-the-art results spanning through various fields. However, a clear and profound theoretical understanding of the forward pass, the core algorithm of CNN, is still lacking. In parallel, within the wide field of sparse approximation, Convolutional Sparse Coding (CSC) has gained increasing attention in recent years. A theoretical study of this model was recently conducted, establishing it as a reliable and stable alternative to the commonly practiced patch-based processing. Herein, we propose a novel multi-layer model, ML-CSC, in which signals are assumed to emerge from a cascade of CSC layers. This is shown to be tightly connected to CNN, so much so that the forward pass of the CNN is in fact the thresholding pursuit serving the ML-CSC model. This connection brings a fresh view to CNN, as we are able to attribute to this architecture theoretical claims such as uniqueness of the representations throughout the network, and their stable estimation, all guaranteed under simple local sparsity conditions. Lastly, identifying the weaknesses in the above pursuit scheme, we propose an alternative to the forward pass, which is connected to deconvolutional, recurrent and residual networks, and has better theoretical guarantees.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Striving for Simplicity: The All Convolutional Net
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Task-Driven Dictionary Learning
• The Loss Surfaces of Multilayer Networks
• Provable Bounds for Learning Some Deep Representations
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding
• Trainlets: Dictionary Learning in High Dimensions
• Working Locally Thinking Globally - Part II: Stability and Algorithms for Convolutional Sparse Coding

Cited by in corpus (11)

• Deep Image Prior
• Prevalence of Neural Collapse during the terminal phase of deep learning training
• Fully Learnable Deep Wavelet Transform for Unsupervised Monitoring of High-Frequency Time Series
• Convolutional Dictionary Learning: Acceleration and Convergence
• Mathematics of Deep Learning
• Take it in your stride: Do we need striding in CNNs?
• A convolutional plane wave model for sound field reconstruction
• Distributed Sequence Memory of Multidimensional Inputs in Recurrent Networks
• Multi-layer Residual Sparsifying Transform (MARS) Model for Low-dose CT Image Reconstruction
• Convolutional Neural Networks Analyzed via Inverse Problem Theory and Sparse Representations
• Deep Epitome for Unravelling Generalized Hamming Network: A Fuzzy Logic Interpretation of Deep Learning