Adaptive regularization for Lasso models in the context of non-stationary data streams

Abstract

Large scale, streaming datasets are ubiquitous in modern machine learning. Streaming algorithms must be scalable, amenable to incremental training and robust to the presence of non-stationarity. In this work consider the problem of learning ${\ell }_1$ regularized linear models in the context of streaming data. In particular, the focus of this work revolves around how to select the regularization parameter when data arrives sequentially and the underlying distribution is non-stationary (implying the choice of optimal regularization parameter is itself time-varying). We propose a framework through which to infer an adaptive regularization parameter. Our approach employs an ${\ell }_1$ penalty constraint where the corresponding sparsity parameter is iteratively updated via stochastic gradient descent. This serves to reformulate the choice of regularization parameter in a principled framework for online learning. The proposed method is derived for linear regression and subsequently extended to generalized linear models. We validate our approach using simulated and real datasets and present an application to a neuroimaging dataset.

20 pages, 3 figures. arXiv admin note: text overlap with arXiv:1511.02187

References in corpus (7)

• Pathwise coordinate optimization
• The Statistical Analysis of fMRI Data
• Piecewise linear regularized solution paths
• Stability Approach to Regularization Selection (StARS) for High Dimensional Graphical Models
• Regularized Estimation of Piecewise Constant Gaussian Graphical Models: The Group-Fused Graphical Lasso
• LOCO: Distributing Ridge Regression with Random Projections
• Graph embeddings of dynamic functional connectivity reveal discriminative patterns of task engagement in HCP data