Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery

Abstract

Technological innovations have revolutionized the process of scientific research and knowledge discovery. The availability of massive data and challenges from frontiers of research and development have reshaped statistical thinking, data analysis and theoretical studies. The challenges of high-dimensionality arise in diverse fields of sciences and the humanities, ranging from computational biology and health studies to financial engineering and risk management. In all of these fields, variable selection and feature extraction are crucial for knowledge discovery. We first give a comprehensive overview of statistical challenges with high dimensionality in these diverse disciplines. We then approach the problem of variable selection and feature extraction using a unified framework: penalized likelihood methods. Issues relevant to the choice of penalty functions are addressed. We demonstrate that for a host of statistical problems, as long as the dimensionality is not excessively large, we can estimate the model parameters as well as if the best model is known in advance. The persistence property in risk minimization is also addressed. The applicability of such a theory and method to diverse statistical problems is demonstrated. Other related problems with high-dimensionality are also discussed.

2 figures

References in corpus (3)

• Nonconcave penalized likelihood with a diverging number of parameters
• Higher criticism for detecting sparse heterogeneous mixtures
• Parametric Inference for Biological Sequence Analysis

Cited by in corpus (40)

• On the "degrees of freedom" of the lasso
• On the adaptive elastic-net with a diverging number of parameters
• One-step sparse estimates in nonconcave penalized likelihood models
• Composite quantile regression and the oracle Model Selection Theory
• High-dimensional classification using features annealed independence rules
• Robust rank correlation based screening
• New efficient estimation and variable selection methods for semiparametric varying-coefficient partially linear models
• A unified approach to model selection and sparse recovery using regularized least squares
• High Dimensional Statistical Inference and Random Matrices
• Estimation and variable selection for generalized additive partial linear models
• GEE analysis of clustered binary data with diverging number of covariates
• The asymptotic distribution and Berry--Esseen bound of a new test for independence in high dimension with an application to stochastic optimization
• Consistent selection of tuning parameters via variable selection stability
• Extending Statistical Boosting - An Overview of Recent Methodological Developments
• Testing for Independence of Large Dimensional Vectors
• A Selective Overview of Variable Selection in High Dimensional Feature Space (Invited Review Article)
• When do stepwise algorithms meet subset selection criteria?
• Rasch-based high-dimensionality data reduction and class prediction with applications to microarray gene expression data
• Impacts of high dimensionality in finite samples
• A ROAD to Classification in High Dimensional Space
• Rejoinder: One-step sparse estimates in nonconcave penalized likelihood models
• Sure Independence Screening for Ultra-High Dimensional Feature Space
• To how many simultaneous hypothesis tests can normal, Student's t or bootstrap calibration be applied?
• Feature selection when there are many influential features
• DropCluster: A structured dropout for convolutional networks
• CoCoLasso for High-dimensional Error-in-variables Regression
• A permutational-splitting sample procedure to quantify expert opinion on clusters of chemical compounds using high-dimensional data
• High Dimensional Covariance Matrix Estimation Using a Factor Model
• High-dimensional varying index coefficient quantile regression model
• Strong Sure Screening of Ultra-high Dimensional Data with Interaction Effects
• Sparse Fisher's Linear Discriminant Analysis for Partially Labeled Data
• A Framework for Challenge Design: Insight and Deployment Challenges to Address Medical Image Analysis Problems
• ET-Lasso: A New Efficient Tuning of Lasso-type Regularization for High-Dimensional Data
• A variable selection approach for highly correlated predictors in high-dimensional genomic data
• Variable Selection with Second-Generation P-Values
• Non-uniform Berry-Esseen Bound by Unbounded Exchangeable Pair Approach
• On the use of cross-validation for the calibration of the adaptive lasso
• An efficient algorithm for structured sparse quantile regression
• Tropical Support Vector Machines: Evaluations and Extension to Function Spaces
• Linear spectral statistics of sequential sample covariance matrices