Sparse Estimation by Exponential Weighting

Abstract

Consider a regression model with fixed design and Gaussian noise where the regression function can potentially be well approximated by a function that admits a sparse representation in a given dictionary. This paper resorts to exponential weights to exploit this underlying sparsity by implementing the principle of sparsity pattern aggregation. This model selection take on sparse estimation allows us to derive sparsity oracle inequalities in several popular frameworks, including ordinary sparsity, fused sparsity and group sparsity. One striking aspect of these theoretical results is that they hold under no condition in the dictionary. Moreover, we describe an efficient implementation of the sparsity pattern aggregation principle that compares favorably to state-of-the-art procedures on some basic numerical examples.

Published in at http://dx.doi.org/10.1214/12-STS393 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (5)

• Nearly unbiased variable selection under minimax concave penalty
• Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection
• Aggregation for Gaussian regression
• Pac-bayesian bounds for sparse regression estimation with exponential weights
• Empirical risk minimization is optimal for the convex aggregation problem

Cited by in corpus (14)

• Bayesian linear regression with sparse priors
• On risk bounds in isotonic and other shape restricted regression problems
• Empirical Bayes posterior concentration in sparse high-dimensional linear models
• Empirical entropy, minimax regret and minimax risk
• PAC-Bayesian Estimation and Prediction in Sparse Additive Models
• Optimal learning with $Q$-aggregation
• Optimal bounds for aggregation of affine estimators
• Aggregation of predictors for nonstationary sub-linear processes and online adaptive forecasting of time varying autoregressive processes
• Comparing and weighting imperfect models using D-probabilities
• From bilinear regression to inductive matrix completion: a quasi-Bayesian analysis
• Optimal exponential bounds for aggregation of density estimators
• Empirical Bayes inference in sparse high-dimensional generalized linear models
• Introduction to the Special Issue on Sparsity and Regularization Methods
• An adaptive multiclass nearest neighbor classifier