Variable selection in nonparametric additive models

Abstract

We consider a nonparametric additive model of a conditional mean function in which the number of variables and additive components may be larger than the sample size but the number of nonzero additive components is "small" relative to the sample size. The statistical problem is to determine which additive components are nonzero. The additive components are approximated by truncated series expansions with B-spline bases. With this approximation, the problem of component selection becomes that of selecting the groups of coefficients in the expansion. We apply the adaptive group Lasso to select nonzero components, using the group Lasso to obtain an initial estimator and reduce the dimension of the problem. We give conditions under which the group Lasso selects a model whose number of components is comparable with the underlying model, and the adaptive group Lasso selects the nonzero components correctly with probability approaching one as the sample size increases and achieves the optimal rate of convergence. The results of Monte Carlo experiments show that the adaptive group Lasso procedure works well with samples of moderate size. A data example is used to illustrate the application of the proposed method.

Published in at http://dx.doi.org/10.1214/09-AOS781 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (23)

• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Nearly unbiased variable selection under minimax concave penalty
• High-dimensional graphs and variable selection with the Lasso
• Nonconcave penalized likelihood with a diverging number of parameters
• Consistency of the group Lasso and multiple kernel learning
• Component selection and smoothing in multivariate nonparametric regression
• Asymptotic properties of bridge estimators in sparse high-dimensional regression models
• The sparsity and bias of the Lasso selection in high-dimensional linear regression
• Lasso-type recovery of sparse representations for high-dimensional data
• High-dimensional additive modeling
• High-dimensional generalized linear models and the lasso
• Sparsity oracle inequalities for the Lasso
• Nonparametric Estimation of an Additive Model With a Link Function
• Discussion: One-step sparse estimates in nonconcave penalized likelihood models

Cited by in corpus (80)

• High-Dimensional Feature Selection by Feature-Wise Kernelized Lasso
• A Selective Review of Group Selection in High-Dimensional Models
• Partially linear additive quantile regression in ultra-high dimension
• Functional additive regression
• Spike-and-Slab Priors for Function Selection in Structured Additive Regression Models
• Feature Selection with Annealing for Computer Vision and Big Data Learning
• Estimation and model selection in generalized additive partial linear models for correlated data with diverging number of covariates
• Group Lasso for high dimensional sparse quantile regression models
• Variable selection in functional regression models: a review
• Local independence feature screening for nonparametric and semiparametric models by marginal empirical likelihood
• Forecasting: theory and practice
• Variable selection and sensitivity analysis using dynamic trees, with an application to computer code performance tuning
• Estimation and inference in generalized additive coefficient models for nonlinear interactions with high-dimensional covariates
• Group Sparse Additive Models
• Doubly Robust Sure Screening for Elliptical Copula Regression Model
• Broadcasted Nonparametric Tensor Regression
• Adaptive Bayesian density regression for high-dimensional data
• SLANTS: Sequential Adaptive Nonlinear Modeling of Vector Time Series
• Nonparametric regression with the scale depending on auxiliary variable
• On Deep Instrumental Variables Estimate
• Semiparametric zero-inflated modeling in multi-ethnic study of atherosclerosis (MESA)
• Improved Sample Complexity for Stochastic Compositional Variance Reduced Gradient
• Ultra High-Dimensional Nonlinear Feature Selection for Big Biological Data
• The restricted consistency property of leave-$n_v$-out cross-validation for high-dimensional variable selection
• Pathwise Coordinate Optimization for Sparse Learning: Algorithm and Theory
• Reduced-rank Regression in Sparse Multivariate Varying-Coefficient Models with High-dimensional Covariates
• Nonparametric Independence Screening via Favored Smoothing Bandwidth
• Functional Group Bridge for Simultaneous Regression and Support Estimation
• Two-step estimation of high dimensional additive models
• Robust nonparametric integrative analysis to decipher heterogeneity and commonality across subgroups using sparse boosting
• A Hierarchical Integrative Group LASSO (HiGLASSO) Framework for Analyzing Environmental Mixtures
• Nonparametric Independence Screening in Sparse Ultra-High Dimensional Additive Models
• Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functions
• Oracle Inequalities and Optimal Inference under Group Sparsity
• Sparse Model Identification and Learning for Ultra-high-dimensional Additive Partially Linear Models
• Estimation and Uniform Inference in Sparse High-Dimensional Additive Models
• Learning Sparse Additive Models with Interactions in High Dimensions
• On Hodges' Superefficiency and Merits of Oracle Property in Model Selection
• Spike-and-Slab LASSO Generalized Additive Models and Scalable Algorithms for High-Dimensional Data Analysis
• Variable selection in high-dimensional additive models based on norms of projections
• Faithful Variable Screening for High-Dimensional Convex Regression
• Momentum with Variance Reduction for Nonconvex Composition Optimization
• N${}^3$LARS: Minimum Redundancy Maximum Relevance Feature Selection for Large and High-dimensional Data
• Ultra high dimensional generalized additive model: Unified Theory and Methods
• Uncertainty Quantification for High Dimensional Sparse Nonparametric Additive Models
• Penalized Likelihood and Bayesian Function Selection in Regression Models
• Flexible Shrinkage Estimation in High-Dimensional Varying Coefficient Models
• Human spatial dynamics for electricity demand forecasting: the case of France during the 2022 energy crisis
• A theory of nonparametric regression in the presence of complex nuisance components
• Grouped Variable Selection via Nested Spike and Slab Priors
• Selection Consistency of EBIC for GLIM with Non-canonical Links and Diverging Number of Parameters
• Sparse Tensor Additive Regression
• SNAP: A semismooth Newton algorithm for pathwise optimization with optimal local convergence rate and oracle properties
• Valid uncertainty quantification about the model in a linear regression setting
• Endogenous Treatment Effect Estimation with some Invalid and Irrelevant Instruments
• LASSO ISOtone for High Dimensional Additive Isotonic Regression
• Nonparametric Variable Screening with Optimal Decision Stumps
• Detecting Nonlinear Causality in Multivariate Time Series with Sparse Additive Models
• Scalable model selection for spatial additive mixed modeling: application to crime analysis
• Greedy Forward Regression for Variable Screening
• Bayesian Quantile Regression for Partially Linear Additive Models
• Dynamic Large Spatial Covariance Matrix Estimation in Application to Semiparametric Model Construction via Variable Clustering: the SCE approach
• Statistical Inference on Partially Linear Panel Model under Unobserved Linearity
• Functional Horseshoe Priors for Subspace Shrinkage
• Model-Free Sure Screening via Maximum Correlation
• Variable Grouping Based Bayesian Additive Regression Tree
• Monotone Splines Lasso
• Improved Oracle Complexity of Variance Reduced Methods for Nonsmooth Convex Stochastic Composition Optimization
• Structure learning via unstructured kernel-based M-regression
• Nonparametric Model Checking and Variable Selection
• Semiparametric Bayesian Information Criterion for Model Selection in Ultra-high Dimensional Additive Models
• Group Spike and Slab Variational Bayes
• High-Dimensional Bayesian Inference in Nonparametric Additive Models
• The ET Interview: Professor Joel L. Horowitz
• Bayesian model selection in additive partial linear models via locally adaptive splines
• A General Framework of Nonparametric Feature Selection in High-Dimensional Data
• Adaptive Estimation In High-Dimensional Additive Models With Multi-Resolution Group Lasso
• Predicting Census Survey Response Rates With Parsimonious Additive Models and Structured Interactions
• Estimation and Model Identification of Locally Stationary Varying-Coefficient Additive Models
• A sure independence screening procedure for ultra-high dimensional partially linear additive models