Structured Sparsity Inducing Adaptive Optimizers for Deep Learning
arXiv:2102.03869
Abstract
The parameters of a neural network are naturally organized in groups, some of which might not contribute to its overall performance. To prune out unimportant groups of parameters, we can include some non-differentiable penalty to the objective function, and minimize it using proximal gradient methods. In this paper, we derive the weighted proximal operator, which is a necessary component of these proximal methods, of two structured sparsity inducing penalties. Moreover, they can be approximated efficiently with a numerical solver, and despite this approximation, we prove that existing convergence guarantees are preserved when these operators are integrated as part of a generic adaptive proximal method. Finally, we show that this adaptive method, together with the weighted proximal operators derived here, is indeed capable of finding solutions with structure in their sparsity patterns, on representative examples from computer vision and natural language processing.
References in corpus (8)
- Nearly unbiased variable selection under minimax concave penalty
- MADE: Masked Autoencoder for Distribution Estimation
- Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
- Reweighted Proximal Pruning for Large-Scale Language Representation
- Learning Neural Causal Models from Unknown Interventions
- Non-asymptotic Analysis of Stochastic Methods for Non-Smooth Non-Convex Regularized Problems
- Proximal Adam: Robust Adaptive Update Scheme for Constrained Optimization
- A General Family of Stochastic Proximal Gradient Methods for Deep Learning