Maximal Sparsity with Deep Networks?

Abstract

The iterations of many sparse estimation algorithms are comprised of a fixed linear filter cascaded with a thresholding nonlinearity, which collectively resemble a typical neural network layer. Consequently, a lengthy sequence of algorithm iterations can be viewed as a deep network with shared, hand-crafted layer weights. It is therefore quite natural to examine the degree to which a learned network model might act as a viable surrogate for traditional sparse estimation in domains where ample training data is available. While the possibility of a reduced computational budget is readily apparent when a ceiling is imposed on the number of layers, our work primarily focuses on estimation accuracy. In particular, it is well-known that when a signal dictionary has coherent columns, as quantified by a large RIP constant, then most tractable iterative algorithms are unable to find maximally sparse representations. In contrast, we demonstrate both theoretically and empirically the potential for a trained deep network to recover minimal ${\ell }_0$-norm representations in regimes where existing methods fail. The resulting system is deployed on a practical photometric stereo estimation problem, where the goal is to remove sparse outliers that can disrupt the estimation of surface normals from a 3D scene.

Cited by in corpus (32)

• Deep Convolutional Neural Network for Inverse Problems in Imaging
• Denoising Prior Driven Deep Neural Network for Image Restoration
• AMP-Inspired Deep Networks for Sparse Linear Inverse Problems
• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI
• Learning to Optimize: A Primer and A Benchmark
• Deep Learning Methods for Solving Linear Inverse Problems: Research Directions and Paradigms
• RE-MIMO: Recurrent and Permutation Equivariant Neural MIMO Detection
• Implicit differentiation of Lasso-type models for hyperparameter optimization
• Deep Coupled-Representation Learning for Sparse Linear Inverse Problems with Side Information
• Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection
• Spatio-Temporal Structured Sparse Regression with Hierarchical Gaussian Process Priors
• Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
• Estimation of Tissue Microstructure Using a Deep Network Inspired by a Sparse Reconstruction Framework
• Improving Person Re-identification with Iterative Impression Aggregation
• Optimization Algorithm Inspired Deep Neural Network Structure Design
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• When and How Can Deep Generative Models be Inverted?
• ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing
• Adversarial Robustness of Supervised Sparse Coding
• Learnable Descent Algorithm for Nonsmooth Nonconvex Image Reconstruction
• Learning step sizes for unfolded sparse coding
• Random mesh projectors for inverse problems
• Deep Learning Based Fast Multiuser Detection for Massive Machine-Type Communication
• A Design Space Study for LISTA and Beyond
• Learning to compress and search visual data in large-scale systems
• Differentiable Greedy Networks
• Learned Interpretable Residual Extragradient ISTA for Sparse Coding
• Data-driven Estimation of Sinusoid Frequencies
• Learning to solve TV regularized problems with unrolled algorithms
• Tree Search Network for Sparse Regression
• Deep Double Sparsity Encoder: Learning to Sparsify Not Only Features But Also Parameters