Screening for Sparse Online Learning
arXiv:2101.06982
Abstract
Sparsity promoting regularizers are widely used to impose low-complexity structure (e.g. l1-norm for sparsity) to the regression coefficients of supervised learning. In the realm of deterministic optimization, the sequence generated by iterative algorithms (such as proximal gradient descent) exhibit "finite activity identification", namely, they can identify the low-complexity structure in a finite number of iterations. However, most online algorithms (such as proximal stochastic gradient descent) do not have the property owing to the vanishing step-size and non-vanishing variance. In this paper, by combining with a screening rule, we show how to eliminate useless features of the iterates generated by online algorithms, and thereby enforce finite activity identification. One consequence is that when combined with any convergent online algorithm, sparsity properties imposed by the regularizer can be exploited for computational gains. Numerically, significant acceleration can be obtained.
References in corpus (10)
- Lasso Screening Rules via Dual Polytope Projection
- Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems
- Screening Tests for Lasso Problems
- Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization
- Geometric properties of solutions to the total variation denoising problem
- Safe Screening With Variational Inequalities and Its Application to LASSO
- A Safe Screening Rule for Sparse Logistic Regression
- GAP Safe screening rules for sparse multi-task and multi-class models
- Local Convergence Properties of SAGA/Prox-SVRG and Acceleration
- Safe Screening for the Generalized Conditional Gradient Method