High-dimensional additive modeling

Abstract

We propose a new sparsity-smoothness penalty for high-dimensional generalized additive models. The combination of sparsity and smoothness is crucial for mathematical theory as well as performance for finite-sample data. We present a computationally efficient algorithm, with provable numerical convergence properties, for optimizing the penalized likelihood. Furthermore, we provide oracle results which yield asymptotic optimality of our estimator for high dimensional but sparse additive models. Finally, an adaptive version of our sparsity-smoothness penalized approach yields large additional performance gains.

Published in at http://dx.doi.org/10.1214/09-AOS692 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (9)

• High-dimensional graphs and variable selection with the Lasso
• Simultaneous analysis of Lasso and Dantzig selector
• Boosting Algorithms: Regularization, Prediction and Model Fitting
• Component selection and smoothing in multivariate nonparametric regression
• The sparsity and bias of the Lasso selection in high-dimensional linear regression
• Lasso-type recovery of sparse representations for high-dimensional data
• High-dimensional generalized linear models and the lasso
• Sparsity oracle inequalities for the Lasso
• Variable selection in high-dimensional linear models: partially faithful distributions and the PC-simple algorithm

Cited by in corpus (96)

• Variable selection in nonparametric additive models
• A Selective Review of Group Selection in High-Dimensional Models
• A Unified Framework for High-Dimensional Analysis of M-Estimators with Decomposable Regularizers
• CAM: Causal additive models, high-dimensional order search and penalized regression
• Sparsity in multiple kernel learning
• Correlated variables in regression: clustering and sparse estimation
• Functional additive regression
• Taking Advantage of Sparsity in Multi-Task Learning
• Spike-and-Slab Priors for Function Selection in Structured Additive Regression Models
• Feature Selection with Annealing for Computer Vision and Big Data Learning
• Exact Post Model Selection Inference for Marginal Screening
• Fast learning rate of multiple kernel learning: Trade-off between sparsity and smoothness
• Generalized Additive Model Selection
• Structured Sparsity and Generalization
• Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification
• Minimax-optimal nonparametric regression in high dimensions
• The Multivariate Hawkes Process in High Dimensions: Beyond Mutual Excitation
• Group Lasso for high dimensional sparse quantile regression models
• Variable selection in functional regression models: a review
• COBRA: A Combined Regression Strategy
• Gradient-based Optimization for Regression in the Functional Tensor-Train Format
• State-of-the-art in selection of variables and functional forms in multivariable analysis -- outstanding issues
• Inference of High-dimensional Autoregressive Generalized Linear Models
• PAC-Bayesian Estimation and Prediction in Sparse Additive Models
• Estimation and inference in generalized additive coefficient models for nonlinear interactions with high-dimensional covariates
• Group Sparse Additive Models
• Minimax-optimal rates for sparse additive models over kernel classes via convex programming
• Doubly Robust Sure Screening for Elliptical Copula Regression Model
• Broadcasted Nonparametric Tensor Regression
• PAC-Bayesian aggregation and multi-armed bandits
• Learning Heteroscedastic Models by Convex Programming under Group Sparsity
• Structured Estimation in Nonparameteric Cox Model
• Sparse Additive Functional and Kernel CCA
• Time Series Featurization via Topological Data Analysis
• The Group Square-Root Lasso: Theoretical Properties and Fast Algorithms
• Impact of subsampling and pruning on random forests
• Nonparametric Regression with Adaptive Truncation via a Convex Hierarchical Penalty
• Nonparametric Independence Screening via Favored Smoothing Bandwidth
• Two-step estimation of high dimensional additive models
• Robust Estimation of Heterogeneous Treatment Effects using Electronic Health Record Data
• Nonparametric Independence Screening in Sparse Ultra-High Dimensional Additive Models
• Binarsity: a penalization for one-hot encoded features in linear supervised learning
• Nonparametric Screening under Conditional Strictly Convex Loss for Ultrahigh Dimensional Sparse Data
• Minimax Optimal Rates of Estimation in High Dimensional Additive Models: Universal Phase Transition
• Learning Sparse Additive Models with Interactions in High Dimensions
• Interpretable Learning-to-Rank with Generalized Additive Models
• Asymptotics for penalized splines in generalized additive models
• Regularizing Double Machine Learning in Partially Linear Endogenous Models
• Estimation and Uniform Inference in Sparse High-Dimensional Additive Models
• RANK: Large-Scale Inference with Graphical Nonlinear Knockoffs
• Variable selection in high-dimensional additive models based on norms of projections
• Mixtures of g-Priors for Generalised Additive Model Selection with Penalised Splines
• How Many Machines Can We Use in Parallel Computing for Kernel Ridge Regression?
• Ultra high dimensional generalized additive model: Unified Theory and Methods
• P-splines with an l1 penalty for repeated measures
• Learning Non-Parametric Basis Independent Models from Point Queries via Low-Rank Methods
• An Algorithm for Nonlinear, Nonparametric Model Choice and Prediction
• Penalized Likelihood and Bayesian Function Selection in Regression Models
• Fused Lasso Additive Model
• Flexible Shrinkage Estimation in High-Dimensional Varying Coefficient Models
• Invariances of random fields paths, with applications in Gaussian Process Regression
• Irrational Exuberance: Correcting Bias in Probability Estimates
• Uncertainty Quantification for High Dimensional Sparse Nonparametric Additive Models
• Random forests and kernel methods
• Fast Convergence Rate of Multiple Kernel Learning with Elastic-net Regularization
• Sparse Tensor Additive Regression
• A theory of nonparametric regression in the presence of complex nuisance components
• Partially Observable Online Change Detection via Smooth-Sparse Decomposition
• A reluctant additive model framework for interpretable nonlinear individualized treatment rules
• Bayesian Quantile Regression for Partially Linear Additive Models
• (Decision and regression) tree ensemble based kernels for regression and classification
• Hierarchical Total Variations and Doubly Penalized ANOVA Modeling for Multivariate Nonparametric Regression
• Extreme Eigenvalues of Nonlinear Correlation Matrices with Applications to Additive Models
• Dimension reduction and variable selection in case control studies via regularized likelihood optimization
• Dynamic Large Spatial Covariance Matrix Estimation in Application to Semiparametric Model Construction via Variable Clustering: the SCE approach
• Component Selection in the Additive Regression Model
• LASSO ISOtone for High Dimensional Additive Isotonic Regression
• Generalized Fiducial Inference for Ultrahigh Dimensional Regression
• Reluctant generalized additive modeling
• Feature Augmentation via Nonparametrics and Selection (FANS) in High Dimensional Classification
• Risk upper bounds for RKHS ridge group sparse estimator in the regression model with non-Gaussian and non-bounded error
• Adaptive Estimation In High-Dimensional Additive Models With Multi-Resolution Group Lasso
• A Framework for an Assessment of the Kernel-target Alignment in Tree Ensemble Kernel Learning
• Nonparametric inference for additive models estimated via simplified smooth backfitting
• Subgroup Analysis for Longitudinal data via Semiparametric Additive Mixed Effect Model
• Predicting Census Survey Response Rates With Parsimonious Additive Models and Structured Interactions
• A Theoretical Framework for Bayesian Nonparametric Regression
• Functional Horseshoe Priors for Subspace Shrinkage
• Nonlinear Mixed-effects Scalar-on-function Models and Variable Selection for Kinematic Upper Limb Movement Data
• A sure independence screening procedure for ultra-high dimensional partially linear additive models
• On the uniform convergence of empirical norms and inner products, with application to causal inference
• Monotone Splines Lasso
• High-Dimensional Bayesian Inference in Nonparametric Additive Models
• Robust Lasso with missing and grossly corrupted observations
• Sharp Convergence Rate and Support Consistency of Multiple Kernel Learning with Sparse and Dense Regularization
• Fast Learning Rate of lp-MKL and its Minimax Optimality