Markov Neighborhood Regression for High-Dimensional Inference

Abstract

This paper proposes an innovative method for constructing confidence intervals and assessing p-values in statistical inference for high-dimensional linear models. The proposed method has successfully broken the high-dimensional inference problem into a series of low-dimensional inference problems: For each regression coefficient ${\beta }_i$, the confidence interval and $p$-value are computed by regressing on a subset of variables selected according to the conditional independence relations between the corresponding variable $X_i$ and other variables. Since the subset of variables forms a Markov neighborhood of $X_i$ in the Markov network formed by all the variables $X_1,X_2,\dots ,X_p$, the proposed method is coined as Markov neighborhood regression. The proposed method is tested on high-dimensional linear, logistic and Cox regression. The numerical results indicate that the proposed method significantly outperforms the existing ones. Based on the Markov neighborhood regression, a method of learning causal structures for high-dimensional linear models is proposed and applied to identification of drug sensitive genes and cancer driver genes. The idea of using conditional independence relations for dimension reduction is general and potentially can be extended to other high-dimensional or big data problems as well.

37 pages, 5 figures

References in corpus (13)

• Nearly unbiased variable selection under minimax concave penalty
• Nonconcave penalized likelihood with a diverging number of parameters
• On the conditions used to prove oracle results for the Lasso
• Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
• Sure independence screening in generalized linear models with NP-dimensionality
• Valid post-selection inference
• Causal Inference and Causal Explanation with Background Knowledge
• Optimal Inference After Model Selection
• Statistical significance in high-dimensional linear models
• Uniform Post Selection Inference for LAD Regression and Other Z-estimation problems
• Variable selection in high-dimensional linear models: partially faithful distributions and the PC-simple algorithm
• Sure Screening for Gaussian Graphical Models
• Statistical inference for high dimensional regression via Constrained Lasso