Orthant Based Proximal Stochastic Gradient Method for -Regularized Optimization
arXiv:2004.03639
Abstract
Sparsity-inducing regularization problems are ubiquitous in machine learning applications, ranging from feature selection to model compression. In this paper, we present a novel stochastic method -- Orthant Based Proximal Stochastic Gradient Method (OBProx-SG) -- to solve perhaps the most popular instance, i.e., the l1-regularized problem. The OBProx-SG method contains two steps: (i) a proximal stochastic gradient step to predict a support cover of the solution; and (ii) an orthant step to aggressively enhance the sparsity level via orthant face projection. Compared to the state-of-the-art methods, e.g., Prox-SG, RDA and Prox-SVRG, the OBProx-SG not only converges to the global optimal solutions (in convex scenario) or the stationary points (in non-convex scenario), but also promotes the sparsity of the solutions substantially. Particularly, on a large number of convex problems, OBProx-SG outperforms the existing methods comprehensively in the aspect of sparsity exploration and objective values. Moreover, the experiments on non-convex deep neural networks, e.g., MobileNetV1 and ResNet18, further demonstrate its superiority by achieving the solutions of much higher sparsity without sacrificing generalization accuracy.
Accepted by ECML 2020
References in corpus (8)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Recurrent Neural Network Regularization
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
- Recovery Guarantees for One-hidden-layer Neural Networks
- A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization
- A Second-Order Method for Convex -Regularized Optimization with Active Set Prediction