Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding

Abstract

The problem of $L_p$-norm constrained coding is to convert signal into code that lies inside an $L_p$-ball and most faithfully reconstructs the signal. Previous works under the name of sparse coding considered the cases of $L_0$ and $L_1$ norms. The cases with $p>1$ values, i.e. non-sparse coding studied in this paper, remain a difficulty. We propose an interpretable deep structure namely Frank-Wolfe Network (F-W Net), whose architecture is inspired by unrolling and truncating the Frank-Wolfe algorithm for solving an $L_p$-norm constrained problem with $p\ge 1$. We show that the Frank-Wolfe solver for the $L_p$-norm constraint leads to a novel closed-form nonlinear unit, which is parameterized by $p$ and termed $poo{l}_p$. The $poo{l}_p$ unit links the conventional pooling, activation, and normalization operations, making F-W Net distinct from existing deep networks either heuristically designed or converted from projected gradient descent algorithms. We further show that the hyper-parameter $p$ can be made learnable instead of pre-chosen in F-W Net, which gracefully solves the non-sparse coding problem even with unknown $p$. We evaluate the performance of F-W Net on an extensive range of simulations as well as the task of handwritten digit recognition, where F-W Net exhibits strong learning capability. We then propose a convolutional version of F-W Net, and apply the convolutional F-W Net into image denoising and super-resolution tasks, where F-W Net all demonstrates impressive effectiveness, flexibility, and robustness.

arXiv commentAccepted to IEEE Transactions on Circuits and Systems for Video Technology. Code and pretrained models: https://github.com/sunke123/FW-Net

References in corpus (26)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Conditional Random Fields as Recurrent Neural Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• PCANet: A Simple Deep Learning Baseline for Image Classification?
• The Convex Geometry of Linear Inverse Problems
• Learning Activation Functions to Improve Deep Neural Networks
• Density Modeling of Images using a Generalized Normalization Transformation
• Fully Connected Deep Structured Networks
• Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds
• Learned Convolutional Sparse Coding
• Learning Deep Structured Models
• Maximal Sparsity with Deep Networks?
• Learning optimal nonlinearities for iterative thresholding algorithms
• Uncertainty Principles and Vector Quantization
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery
• Learnable Pooling Regions for Image Classification
• Democratic Representations
• A Unified Optimization View on Generalized Matching Pursuit and Frank-Wolfe
• Randomized Block Frank-Wolfe for Convergent Large-Scale Learning
• Learning Deep ${\ell }_0$ Encoders
• Understanding Trainable Sparse Coding via Matrix Factorization
• Learning A Deep ${\ell }_{\infty }$ Encoder for Hashing

Cited by in corpus (2)

• Learning to Optimize: A Primer and A Benchmark
• Hybrid ISTA: Unfolding ISTA With Convergence Guarantees Using Free-Form Deep Neural Networks