Boosting Algorithms: Regularization, Prediction and Model Fitting

Abstract

We present a statistical perspective on boosting. Special emphasis is given to estimating potentially complex parametric or nonparametric models, including generalized linear and additive models as well as regression models for survival analysis. Concepts of degrees of freedom and corresponding Akaike or Bayesian information criteria, particularly useful for regularization and variable selection in high-dimensional covariate spaces, are discussed as well. The practical aspects of boosting procedures for fitting statistical models are illustrated by means of the dedicated open-source software package mboost. This package implements functions which can be used for model fitting, prediction and variable selection. It is flexible, allowing for the implementation of new boosting algorithms optimizing user-specified loss functions.

This paper commented in: [arXiv:0804.2757], [arXiv:0804.2770]. Rejoinder in [arXiv:0804.2777]. Published in at http://dx.doi.org/10.1214/07-STS242 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (14)

• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• High-dimensional graphs and variable selection with the Lasso
• Asymptotic properties of bridge estimators in sparse high-dimensional regression models
• Boosting for high-dimensional linear models
• Boosting with early stopping: Convergence and consistency

Cited by in corpus (29)

• To Explain or to Predict?
• High-dimensional additive modeling
• The Evolution of Boosting Algorithms - From Machine Learning to Statistical Modelling
• CAM: Causal additive models, high-dimensional order search and penalized regression
• Large-scale benchmark study of survival prediction methods using multi-omics data
• Boosting the concordance index for survival data - a unified framework to derive and evaluate biomarker combinations
• Spike-and-Slab Priors for Function Selection in Structured Additive Regression Models
• Boosting algorithms in energy research: A systematic review
• New multicategory boosting algorithms based on multicategory Fisher-consistent losses
• Stability selection for component-wise gradient boosting in multiple dimensions
• Extending Statistical Boosting - An Overview of Recent Methodological Developments
• A Unified Framework of Constrained Regression
• Small area estimation of the homeless in Los Angeles: An application of cost-sensitive stochastic gradient boosting
• Predicting time to graduation at a large enrollment American university
• A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-${\ell }_1$-Norm Interpolated Classifiers
• Probabilistic water demand forecasting using quantile regression algorithms
• Empirical risk minimization is optimal for the convex aggregation problem
• Risperidone response in patients with schizophrenia drives DNA methylation changes in immune and neuronal systems
• Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble
• Robust nonparametric integrative analysis to decipher heterogeneity and commonality across subgroups using sparse boosting
• An integrative sparse boosting analysis of cancer genomic commonality and difference
• Survival ensembles by the sum of pairwise differences with application to lung cancer microarray studies
• Boosted nonparametric hazards with time-dependent covariates
• Roster Evaluation Based on Classifiers for the Nurse Rostering Problem
• A review of machine learning concepts and methods for addressing challenges in probabilistic hydrological post-processing and forecasting
• Marginal integration for nonparametric causal inference
• Boosting in Univariate Nonparametric Maximum Likelihood Estimation
• Clustering of variables for enhanced interpretability of predictive models
• Boosting prediction with data missing not at random