Component selection and smoothing in multivariate nonparametric regression

Abstract

We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The ``COSSO'' is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.

Published at http://dx.doi.org/10.1214/009053606000000722 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (10)

• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al

Cited by in corpus (78)

• Variable selection in nonparametric additive models
• High-dimensional additive modeling
• Breaking the Curse of Dimensionality with Convex Neural Networks
• A Selective Review of Group Selection in High-Dimensional Models
• Least angle and ${\ell }_1$ penalized regression: A review
• Sparsity in multiple kernel learning
• Group-Lasso on Splines for Spectrum Cartography
• Spike-and-Slab Priors for Function Selection in Structured Additive Regression Models
• Bayesian group Lasso for nonparametric varying-coefficient models with application to functional genome-wide association studies
• Transposable regularized covariance models with an application to missing data imputation
• Nonparametric sparsity and regularization
• High-Dimensional Non-Linear Variable Selection through Hierarchical Kernel Learning
• Penalized variable selection procedure for Cox models with semiparametric relative risk
• Machine Learning Methods in the Computational Biology of Cancer
• Local independence feature screening for nonparametric and semiparametric models by marginal empirical likelihood
• Group Sparse Additive Models
• Minimax-optimal rates for sparse additive models over kernel classes via convex programming
• Nonparametric Basis Pursuit via Sparse Kernel-based Learning
• Broadcasted Nonparametric Tensor Regression
• Learning Heteroscedastic Models by Convex Programming under Group Sparsity
• HARFE: Hard-Ridge Random Feature Expansion
• Structured Estimation in Nonparameteric Cox Model
• Learning to Diversify via Weighted Kernels for Classifier Ensemble
• A Unified Approach to Error Bounds for Structured Convex Optimization Problems
• Variable Selection with ABC Bayesian Forests
• A Hierarchical Integrative Group LASSO (HiGLASSO) Framework for Analyzing Environmental Mixtures
• Two-step estimation of high dimensional additive models
• Group Variable Selection via a Hierarchical Lasso and Its Oracle Property
• Sparse Additive Models
• Nonparametric Independence Screening in Sparse Ultra-High Dimensional Additive Models
• Minimax Optimal Rates of Estimation in High Dimensional Additive Models: Universal Phase Transition
• Estimation and Uniform Inference in Sparse High-Dimensional Additive Models
• Interpretable Learning-to-Rank with Generalized Additive Models
• An analysis of penalized interaction models
• Structured nonlinear variable selection
• Flexible Shrinkage Estimation in High-Dimensional Varying Coefficient Models
• P-splines with an l1 penalty for repeated measures
• Fused Lasso Additive Model
• The Knowledge Gradient Policy Using A Sparse Additive Belief Model
• Sparse Partially Linear Additive Models
• Learning Non-Parametric Basis Independent Models from Point Queries via Low-Rank Methods
• Modelling Interactions in High-dimensional Data with Backtracking
• Shrinkage Estimation and Selection for Multiple Functional Regression
• Double Sparsity Kernel Learning with Automatic Variable Selection and Data Extraction
• A reluctant additive model framework for interpretable nonlinear individualized treatment rules
• Clustering and Variable Selection in the Presence of Mixed Variable Types and Missing Data
• Residual Weighted Learning for Estimating Individualized Treatment Rules
• spikeSlabGAM: Bayesian Variable Selection, Model Choice and Regularization for Generalized Additive Mixed Models in R
• Post-Regularization Confidence Bands for Ordinary Differential Equations
• Factor Importance Ranking and Selection using Total Indices
• High-Dimensional Semiparametric Selection Models: Estimation Theory with an Application to the Retail Gasoline Market
• Efficient Learning of Optimal Individualized Treatment Rules for Heteroscedastic or Misspecified Treatment-Free Effect Models
• Reluctant generalized additive modeling
• Hierarchical Total Variations and Doubly Penalized ANOVA Modeling for Multivariate Nonparametric Regression
• Component Selection in the Additive Regression Model
• Detecting Nonlinear Causality in Multivariate Time Series with Sparse Additive Models
• The problem of perfect predictors in statistical spike train models
• High-Dimensional Bayesian Inference in Nonparametric Additive Models
• KNIFE: Kernel Iterative Feature Extraction
• Learning sparse gradients for variable selection and dimension reduction
• A sure independence screening procedure for ultra-high dimensional partially linear additive models
• Predicting Census Survey Response Rates With Parsimonious Additive Models and Structured Interactions
• The SKIM-FA Kernel: High-Dimensional Variable Selection and Nonlinear Interaction Discovery in Linear Time
• Bayesian model selection in additive partial linear models via locally adaptive splines
• Kernel Ordinary Differential Equations
• Block based refitting in ${\ell }_{12}$ sparse regularisation
• Structure learning via unstructured kernel-based M-regression
• A gradient-based variable selection for binary classification in reproducing kernel Hilbert space
• Adaptive Estimation In High-Dimensional Additive Models With Multi-Resolution Group Lasso
• Learning general sparse additive models from point queries in high dimensions
• Learning rates for the risk of kernel based quantile regression estimators in additive models
• Functional Horseshoe Priors for Subspace Shrinkage
• A General Framework of Nonparametric Feature Selection in High-Dimensional Data
• Nonparametric principal subspace regression
• Estimation of a sparse group of sparse vectors
• A Graphical View of Bayesian Variable Selection
• Sharp Convergence Rate and Support Consistency of Multiple Kernel Learning with Sparse and Dense Regularization
• Sparse additive regression on a regular lattice