Half-Space Proximal Stochastic Gradient Method for Group-Sparsity Regularized Problem
arXiv:2009.12078
Abstract
Optimizing with group sparsity is significant in enhancing model interpretability in machining learning applications, e.g., feature selection, compressed sensing and model compression. However, for large-scale stochastic training problems, effective group sparsity exploration are typically hard to achieve. Particularly, the state-of-the-art stochastic optimization algorithms usually generate merely dense solutions. To overcome this shortage, we propose a stochastic method -- Half-space Stochastic Projected Gradient (HSPG) method to search solutions of high group sparsity while maintain the convergence. Initialized by a simple Prox-SG Step, the HSPG method relies on a novel Half-Space Step to substantially boost the sparsity level. Numerically, HSPG demonstrates its superiority in deep neural networks, e.g., VGG16, ResNet18 and MobileNetV1, by computing solutions of higher group sparsity, competitive objective values and generalization accuracy.
In the future, we might split the work into two separate papers for theoretical analysis and numerical applications on deep learning respectively
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
- Group Sparse Regularization for Deep Neural Networks
- Structured Sparse Principal Component Analysis
- Learning with Structured Sparsity
- SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
- Recovery Guarantees for One-hidden-layer Neural Networks
- A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
- Statistical Adaptive Stochastic Gradient Methods
- Combinatorial Penalties: Which structures are preserved by convex relaxations?