Mirror averaging with sparsity priors

Abstract

We consider the problem of aggregating the elements of a possibly infinite dictionary for building a decision procedure that aims at minimizing a given criterion. Along with the dictionary, an independent identically distributed training sample is available, on which the performance of a given procedure can be tested. In a fairly general set-up, we establish an oracle inequality for the Mirror Averaging aggregate with any prior distribution. By choosing an appropriate prior, we apply this oracle inequality in the context of prediction under sparsity assumption for the problems of regression with random design, density estimation and binary classification.

Published in at http://dx.doi.org/10.3150/11-BEJ361 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

References in corpus (14)

• The sparsity and bias of the Lasso selection in high-dimensional linear regression
• High-dimensional generalized linear models and the lasso
• Sparsity oracle inequalities for the Lasso
• Aggregation for Gaussian regression
• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• The Dantzig selector and sparsity oracle inequalities
• Some sharp performance bounds for least squares regression with $L_1$ regularization
• Fast learning rates in statistical inference through aggregation
• On optimality of Bayesian testimation in the normal means problem
• PAC-Bayesian Bounds for Randomized Empirical Risk Minimizers
• Sparse recovery in convex hulls via entropy penalization
• A universal procedure for aggregating estimators
• Optimal rates of aggregation in classification under low noise assumption
• Sharper lower bounds on the performance of the empirical risk minimization algorithm

Cited by in corpus (11)

• On the Prediction Performance of the Lasso
• Sparse Estimation by Exponential Weighting
• Empirical entropy, minimax regret and minimax risk
• Sharp Oracle Inequalities for Aggregation of Affine Estimators
• Pac-bayesian bounds for sparse regression estimation with exponential weights
• Optimal learning with $Q$-aggregation
• Adaptive Bayesian density regression for high-dimensional data
• Optimal exponential bounds for aggregation of density estimators
• High-dimensional sparse classification using exponential weighting with empirical hinge loss
• Concentration properties of fractional posterior in 1-bit matrix completion
• A Quasi-Bayesian Perspective to Online Clustering