Variable selection using MM algorithms

Abstract

Variable selection is fundamental to high-dimensional statistical modeling. Many variable selection techniques may be implemented by maximum penalized likelihood using various penalty functions. Optimizing the penalized likelihood function is often challenging because it may be nondifferentiable and/or nonconcave. This article proposes a new class of algorithms for finding a maximizer of the penalized likelihood for a broad class of penalty functions. These algorithms operate by perturbing the penalty function slightly to render it differentiable, then optimizing this differentiable function using a minorize-maximize (MM) algorithm. MM algorithms are useful extensions of the well-known class of EM algorithms, a fact that allows us to analyze the local and global convergence of the proposed algorithm using some of the techniques employed for EM algorithms. In particular, we prove that when our MM algorithms converge, they must converge to a desirable point; we also discuss conditions under which this convergence may be guaranteed. We exploit the Newton-Raphson-like aspect of these algorithms to propose a sandwich estimator for the standard errors of the estimators. Our method performs well in numerical tests.

Published at http://dx.doi.org/10.1214/009053605000000200 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (1)

• Nonconcave penalized likelihood with a diverging number of parameters

Cited by in corpus (41)

• Nearly unbiased variable selection under minimax concave penalty
• Challenges of Big Data Analysis
• One-step sparse estimates in nonconcave penalized likelihood models
• Sparse permutation invariant covariance estimation
• Asymptotic properties of bridge estimators in sparse high-dimensional regression models
• A Minorization-Maximization Method for Optimizing Sum Rate in Non-Orthogonal Multiple Access Systems
• Strong oracle optimality of folded concave penalized estimation
• Network exploration via the adaptive LASSO and SCAD penalties
• New efficient estimation and variable selection methods for semiparametric varying-coefficient partially linear models
• Variable selection in semiparametric regression modeling
• A unified approach to model selection and sparse recovery using regularized least squares
• SCAD-penalized regression in high-dimensional partially linear models
• Coauthorship and Citation Networks for Statisticians
• Calibrating nonconvex penalized regression in ultra-high dimension
• Estimation and variable selection for generalized additive partial linear models
• Endogeneity in high dimensions
• Sparse logistic principal components analysis for binary data
• Optimal computational and statistical rates of convergence for sparse nonconvex learning problems
• The MM Alternative to EM
• Variable selection in measurement error models
• On optimality of Bayesian testimation in the normal means problem
• Estimation in high-dimensional linear models with deterministic design matrices
• Nonconcave penalized composite conditional likelihood estimation of sparse Ising models
• Discussion: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$
• Sparse Classification: a scalable discrete optimization perspective
• Global solutions to folded concave penalized nonconvex learning
• A fast algorithm for detecting gene-gene interactions in genome-wide association studies
• Model selection and structure specification in ultra-high dimensional generalised semi-varying coefficient models
• On the estimation of variance parameters in non-standard generalised linear mixed models: Application to penalised smoothing
• MM Algorithms for Minimizing Nonsmoothly Penalized Objective Functions
• Robust Truss Topology Optimization via Semidefinite Programming with Complementarity Constraints: A Difference-of-Convex Programming Approach
• Sparse and Smooth Functional Data Clustering
• Sparse Multinomial Logistic Regression via Approximate Message Passing
• Nonconvex Optimization via MM Algorithms: Convergence Theory
• Penalized regression via the restricted bridge estimator
• Identification of oscillatory brain networks with Hidden Gaussian Graphical Spectral models of EEG/MEG
• MM Algorithms for Distance Covariance based Sufficient Dimension Reduction and Sufficient Variable Selection
• Penalized G-estimation for effect modifier selection in a structural nested mean model for repeated outcomes
• MM for Penalized Estimation
• Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization
• Wilcoxon-type Multivariate Cluster Elastic Net