A lasso for hierarchical interactions

Abstract

We add a set of convex constraints to the lasso to produce sparse interaction models that honor the hierarchy restriction that an interaction only be included in a model if one or both variables are marginally important. We give a precise characterization of the effect of this hierarchy constraint, prove that hierarchy holds with probability one and derive an unbiased estimate for the degrees of freedom of our estimator. A bound on this estimate reveals the amount of fitting "saved" by the hierarchy constraint. We distinguish between parameter sparsity - the number of nonzero coefficients - and practical sparsity - the number of raw variables one must measure to make a new prediction. Hierarchy focuses on the latter, which is more closely tied to important data collection concerns such as cost, time and effort. We develop an algorithm, available in the R package hierNet, and perform an empirical study of our method.

Published in at http://dx.doi.org/10.1214/13-AOS1096 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (5)

• On the "degrees of freedom" of the lasso
• The composite absolute penalties family for grouped and hierarchical variable selection
• Degrees of freedom in lasso problems
• Group Lasso with Overlaps: the Latent Group Lasso approach
• Structured variable selection and estimation

Cited by in corpus (23)

• Supersparse Linear Integer Models for Optimized Medical Scoring Systems
• Hierarchical Sparse Modeling: A Choice of Two Group Lasso Formulations
• Convex Modeling of Interactions with Strong Heredity
• Variable selection for general index models via sliced inverse regression
• An Ordered Lasso and Sparse Time-Lagged Regression
• Iteratively reweighted adaptive lasso for conditional heteroscedastic time series with applications to AR-ARCH type processes
• Prediction of hierarchical time series using structured regularization and its application to artificial neural networks
• On the Sensitivity of the Lasso to the Number of Predictor Variables
• Semiparametric integrative interaction analysis for non-small-cell lung cancer
• Convex hierarchical testing of interactions
• A Hierarchical Integrative Group LASSO (HiGLASSO) Framework for Analyzing Environmental Mixtures
• mpower: An R Package for Power Analysis of Exposure Mixture Studies via Monte Carlo Simulations
• Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis
• An analysis of penalized interaction models
• Parameter Estimation with the Ordered ${\ell }_2$ Regularization via an Alternating Direction Method of Multipliers
• A scalable hierarchical lasso for gene-environment interactions
• An ADMM approach for multi-response regression with overlapping groups and interaction effects
• Approximate Laplace approximations for scalable model selection
• Adaptive Bi-Level Variable Selection of Conditional Main Effects for Generalized Linear Models
• IMMIGRATE: A Margin-based Feature Selection Method with Interaction Terms
• Predicting Census Survey Response Rates With Parsimonious Additive Models and Structured Interactions
• Interaction as Interference: A Quantum-Inspired Aggregation Approach
• Topological Techniques in Model Selection