Learning by mirror averaging

Abstract

Given a finite collection of estimators or classifiers, we study the problem of model selection type aggregation, that is, we construct a new estimator or classifier, called aggregate, which is nearly as good as the best among them with respect to a given risk criterion. We define our aggregate by a simple recursive procedure which solves an auxiliary stochastic linear programming problem related to the original nonlinear one and constitutes a special case of the mirror averaging algorithm. We show that the aggregate satisfies sharp oracle inequalities under some general assumptions. The results are applied to several problems including regression, classification and density estimation.

Published in at http://dx.doi.org/10.1214/07-AOS546 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (4)

• Aggregation for Gaussian regression
• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• Complexity regularization via localized random penalties
• Spatial aggregation of local likelihood estimates with applications to classification

Cited by in corpus (20)

• Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity
• Sparse Regression Learning by Aggregation and Langevin Monte-Carlo
• Fast learning rates in statistical inference through aggregation
• Sparse Estimation by Exponential Weighting
• Empirical entropy, minimax regret and minimax risk
• Deviation optimal learning using greedy Q-aggregation
• Kullback-Leibler aggregation and misspecified generalized linear models
• Mirror averaging with sparsity priors
• Pac-bayesian bounds for sparse regression estimation with exponential weights
• Performance of empirical risk minimization in linear aggregation
• Optimal learning with $Q$-aggregation
• Parameter tuning in pointwise adaptation using a propagation approach
• Optimal rates of aggregation in classification under low noise assumption
• A universal procedure for aggregating estimators
• Empirical risk minimization is optimal for the convex aggregation problem
• Exponential weights in multivariate regression and a low-rankness favoring prior
• On the optimality of the aggregate with exponential weights for low temperatures
• Optimal exponential bounds for aggregation of density estimators
• Discussion of ``2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization'' by V. Koltchinskii
• An adaptive multiclass nearest neighbor classifier