A Novel Framework for Online Supervised Learning with Feature Selection

Abstract

Current online learning methods suffer issues such as lower convergence rates and limited capability to select important features compared to their offline counterparts. In this paper, a novel framework for online learning based on running averages is proposed. Many popular offline regularized methods such as Lasso, Elastic Net, Minimax Concave Penalty (MCP), and Feature Selection with Annealing (FSA) have their online versions introduced in this framework. The equivalence between the proposed online methods and their offline counterparts is proved, and then novel theoretical true support recovery and convergence guarantees are provided for some of the methods in this framework. Numerical experiments indicate that the proposed methods enjoy high true support recovery accuracy and a faster convergence rate compared with conventional online and offline algorithms. Finally, applications to large datasets are presented, where again the proposed framework shows competitive results compared to popular online and offline algorithms.

This version has been accepted by Journal of Nonparametric Statistics

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Nearly unbiased variable selection under minimax concave penalty
• Billion-scale semi-supervised learning for image classification
• Sparse Online Learning via Truncated Gradient
• Online Updating of Statistical Inference in the Big Data Setting
• Better Mini-Batch Algorithms via Accelerated Gradient Methods
• Feature Selection with Annealing for Computer Vision and Big Data Learning
• Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization
• Finite sample change point inference and identification for high-dimensional mean vectors
• Perishability of Data: Dynamic Pricing under Varying-Coefficient Models
• L1-Regularized Least Squares for Support Recovery of High Dimensional Single Index Models with Gaussian Designs