Aggregation for Gaussian regression

Abstract

This paper studies statistical aggregation procedures in the regression setting. A motivating factor is the existence of many different methods of estimation, leading to possibly competing estimators. We consider here three different types of aggregation: model selection (MS) aggregation, convex (C) aggregation and linear (L) aggregation. The objective of (MS) is to select the optimal single estimator from the list; that of (C) is to select the optimal convex combination of the given estimators; and that of (L) is to select the optimal linear combination of the given estimators. We are interested in evaluating the rates of convergence of the excess risks of the estimators obtained by these procedures. Our approach is motivated by recently published minimax results [Nemirovski, A. (2000). Topics in non-parametric statistics. Lectures on Probability Theory and Statistics (Saint-Flour, 1998). Lecture Notes in Math. 1738 85--277. Springer, Berlin; Tsybakov, A. B. (2003). Optimal rates of aggregation. Learning Theory and Kernel Machines. Lecture Notes in Artificial Intelligence 2777 303--313. Springer, Heidelberg]. There exist competing aggregation procedures achieving optimal convergence rates for each of the (MS), (C) and (L) cases separately. Since these procedures are not directly comparable with each other, we suggest an alternative solution. We prove that all three optimal rates, as well as those for the newly introduced (S) aggregation (subset selection), are nearly achieved via a single ``universal'' aggregation procedure. The procedure consists of mixing the initial estimators with weights obtained by penalized least squares. Two different penalties are considered: one of them is of the BIC type, the second one is a data-dependent ${\ell }_1$-type penalty.

Published in at http://dx.doi.org/10.1214/009053606000001587 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (12)

• Rejoinder to "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• Consistent covariate selection and post model selection inference in semiparametric regression

Cited by in corpus (44)

• Simultaneous analysis of Lasso and Dantzig selector
• On the conditions used to prove oracle results for the Lasso
• Least squares after model selection in high-dimensional sparse models
• Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming
• Lasso-type recovery of sparse representations for high-dimensional data
• Estimation of high-dimensional low-rank matrices
• Near-ideal model selection by ${\ell }_1$ minimization
• A Unified Framework for High-Dimensional Analysis of M-Estimators with Decomposable Regularizers
• Sparse recovery under matrix uncertainty
• On the Prediction Performance of the Lasso
• Lasso and probabilistic inequalities for multivariate point processes
• Pivotal estimation via square-root Lasso in nonparametric regression
• Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators
• Some sharp performance bounds for least squares regression with $L_1$ regularization
• Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity
• Sparse Regression Learning by Aggregation and Langevin Monte-Carlo
• Honest variable selection in linear and logistic regression models via ${\ell }_1$ and ${\ell }_1+{\ell }_2$ penalization
• Learning by mirror averaging
• Rejoinder: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$
• Sparse Estimation by Exponential Weighting
• Kullback-Leibler aggregation and misspecified generalized linear models
• Sharp Oracle Inequalities for Aggregation of Affine Estimators
• Simultaneous adaptation to the margin and to complexity in classification
• Mirror averaging with sparsity priors
• SPADES and mixture models
• Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices
• Pac-bayesian bounds for sparse regression estimation with exponential weights
• Sparsity considerations for dependent observations
• Quasi-Likelihood and/or Robust Estimation in High Dimensions
• Optimal learning with $Q$-aggregation
• Performance of empirical risk minimization in linear aggregation
• Discussion: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$
• Optimal rates of aggregation in classification under low noise assumption
• A universal procedure for aggregating estimators
• Empirical risk minimization is optimal for the convex aggregation problem
• Greedy algorithms for prediction
• Robust subset selection
• Structured Estimation in Nonparameteric Cox Model
• On the Sensitivity of the Lasso to the Number of Predictor Variables
• On the optimality of the aggregate with exponential weights for low temperatures
• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• Rejoinder: 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• Performance of Empirical Risk Minimization for Linear Regression with Dependent Data
• Sample average approximation with heavier tails II: localization in stochastic convex optimization and persistence results for the Lasso