Towards Understanding Residual and Dilated Dense Neural Networks via Convolutional Sparse Coding

Abstract

Convolutional neural network (CNN) and its variants have led to many state-of-art results in various fields. However, a clear theoretical understanding about them is still lacking. Recently, multi-layer convolutional sparse coding (ML-CSC) has been proposed and proved to equal such simply stacked networks (plain networks). Here, we think three factors in each layer of it including the initialization, the dictionary design and the number of iterations greatly affect the performance of ML-CSC. Inspired by these considerations, we propose two novel multi-layer models--residual convolutional sparse coding model (Res-CSC) and mixed-scale dense convolutional sparse coding model (MSD-CSC), which have close relationship with the residual neural network (ResNet) and mixed-scale (dilated) dense neural network (MSDNet), respectively. Mathematically, we derive the shortcut connection in ResNet as a special case of a new forward propagation rule on ML-CSC. We find a theoretical interpretation of the dilated convolution and dense connection in MSDNet by analyzing MSD-CSC, which gives a clear mathematical understanding about them. We implement the iterative soft thresholding algorithm (ISTA) and its fast version to solve Res-CSC and MSD-CSC, which can employ the unfolding operation for further improvements. At last, extensive numerical experiments and comparison with competing methods demonstrate their effectiveness using three typical datasets.

arXiv comment13 pages, 8 figures

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• PyCUDA and PyOpenCL: A Scripting-Based Approach to GPU Run-Time Code Generation
• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
• Fast Context Adaptation via Meta-Learning
• Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding
• Memory-Efficient Implementation of DenseNets