Latent variable graphical model selection via convex optimization

Abstract

Suppose we observe samples of a subset of a collection of random variables. No additional information is provided about the number of latent variables, nor of the relationship between the latent and observed variables. Is it possible to discover the number of latent components, and to learn a statistical model over the entire collection of variables? We address this question in the setting in which the latent and observed variables are jointly Gaussian, with the conditional statistics of the observed variables conditioned on the latent variables being specified by a graphical model. As a first step we give natural conditions under which such latent-variable Gaussian graphical models are identifiable given marginal statistics of only the observed variables. Essentially these conditions require that the conditional graphical model among the observed variables is sparse, while the effect of the latent variables is "spread out" over most of the observed variables. Next we propose a tractable convex program based on regularized maximum-likelihood for model selection in this latent-variable setting; the regularizer uses both the ${\ell }_1$ norm and the nuclear norm. Our modeling framework can be viewed as a combination of dimensionality reduction (to identify latent variables) and graphical modeling (to capture remaining statistical structure not attributable to the latent variables), and it consistently estimates both the number of latent components and the conditional graphical model structure among the observed variables. These results are applicable in the high-dimensional setting in which the number of latent/observed variables grows with the number of samples of the observed variables. The geometric properties of the algebraic varieties of sparse matrices and of low-rank matrices play an important role in our analysis.

Published in at http://dx.doi.org/10.1214/11-AOS949 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (6)

• Regularized estimation of large covariance matrices
• Covariance regularization by thresholding
• Sparse permutation invariant covariance estimation
• Operator norm consistent estimation of large-dimensional sparse covariance matrices
• Consistency of trace norm minimization
• High-dimensional covariance estimation by minimizing ${\ell }_1$-penalized log-determinant divergence

Cited by in corpus (115)

• A Tutorial on Regularized Partial Correlation Networks
• Exploratory graph analysis: a new approach for estimating the number of dimensions in psychological research
• Estimating psychopathological networks: be careful what you wish for
• Decomposition into Low-rank plus Additive Matrices for Background/Foreground Separation: A Review for a Comparative Evaluation with a Large-Scale Dataset
• Asymptotic normality and optimalities in estimation of large Gaussian graphical models
• Signal Processing on Graphs: Causal Modeling of Unstructured Data
• High-Dimensional Inference: Confidence Intervals, $p$-Values and R-Software hdi
• Tightness of the maximum likelihood semidefinite relaxation for angular synchronization
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Generalized Symmetric ADMM for Separable Convex Optimization
• Solving Multiple-Block Separable Convex Minimization Problems Using Two-Block Alternating Direction Method of Multipliers
• Low Rank and Structured Modeling of High-dimensional Vector Autoregressions
• Scalable Robust Matrix Recovery: Frank-Wolfe Meets Proximal Methods
• Finding Dense Clusters via "Low Rank + Sparse" Decomposition
• Subspace-Orbit Randomized Decomposition for Low-rank Matrix Approximation
• Parallel Multi-Block ADMM with o(1/k) Convergence
• The achievable performance of convex demixing
• Robust Kronecker Product PCA for Spatio-Temporal Covariance Estimation
• Learning the Dependence Graph of Time Series with Latent Factors
• Adaptive estimation of the copula correlation matrix for semiparametric elliptical copulas
• Clustering using Max-norm Constrained Optimization
• Learning loopy graphical models with latent variables: Efficient methods and guarantees
• Interpolation and Extrapolation of Toeplitz Matrices via Optimal Mass Transport
• Robust Matrix Decomposition with Outliers
• Confounder Adjustment in Multiple Hypothesis Testing
• Learning Latent Variable Gaussian Graphical Models
• Marginal Likelihoods for Distributed Parameter Estimation of Gaussian Graphical Models
• Linear Estimating Equations for Exponential Families with Application to Gaussian Linear Concentration Models
• A Proximal Approach for a Class of Matrix Optimization Problems
• Parallel Direction Method of Multipliers
• Fast Algorithms for Demixing Sparse Signals from Nonlinear Observations
• Spectral Deconfounding via Perturbed Sparse Linear Models
• Local Tomography of Large Networks under the Low-Observability Regime
• SILVar: Single Index Latent Variable Models
• Nonlinear Causal Discovery with Confounders
• Detection of Block-Exchangeable Structure in Large-Scale Correlation Matrices
• Simultaneously Structured Models with Application to Sparse and Low-rank Matrices
• Multi-Scale Factor Analysis of High-Dimensional Brain Signals
• Estimating Differential Latent Variable Graphical Models with Applications to Brain Connectivity
• Fast Proximal Linearized Alternating Direction Method of Multiplier with Parallel Splitting
• Learning Dependency Structures for Weak Supervision Models
• Joint Network Topology Inference in the Presence of Hidden Nodes
• A large covariance matrix estimator under intermediate spikiness regimes
• Recovering Model Structures from Large Low Rank and Sparse Covariance Matrix Estimation
• Ivy: Instrumental Variable Synthesis for Causal Inference
• Sharp recovery bounds for convex demixing, with applications
• Learning a Factor Model via Regularized PCA
• Speeding Up Latent Variable Gaussian Graphical Model Estimation via Nonconvex Optimizations
• A Fused Latent and Graphical Model for Multivariate Binary Data
• Proximal Markov chain Monte Carlo algorithms
• Inference for Individual Mediation Effects and Interventional Effects in Sparse High-Dimensional Causal Graphical Models
• A Unified Framework for Low-Rank plus Sparse Matrix Recovery
• Interpreting Latent Variables in Factor Models via Convex Optimization
• Scalable Interpretable Multi-Response Regression via SEED
• Graph quilting: graphical model selection from partially observed covariances
• Training Complex Models with Multi-Task Weak Supervision
• Alternating Direction Methods for Latent Variable Gaussian Graphical Model Selection
• Network inference in matrix-variate Gaussian models with non-independent noise
• RSVP-graphs: Fast High-dimensional Covariance Matrix Estimation under Latent Confounding
• Tree-based Node Aggregation in Sparse Graphical Models
• Efficient Latent Variable Graphical Model Selection via Split Bregman Method
• Adaptive Estimation in Structured Factor Models with Applications to Overlapping Clustering
• Ising Models with Latent Conditional Gaussian Variables
• Accelerated Alternating Direction Method of Multipliers: an Optimal $O(1/K)$ Nonergodic Analysis
• Bridging Convex and Nonconvex Optimization in Robust PCA: Noise, Outliers, and Missing Data
• Sparse Graphical Linear Dynamical Systems
• Recovering the Graph Underlying Networked Dynamical Systems under Partial Observability: A Deep Learning Approach
• Thresholded Graphical Lasso Adjusts for Latent Variables: Application to Functional Neural Connectivity
• Sparse Cholesky covariance parametrization for recovering latent structure in ordered data
• Latent variable model selection for Gaussian conditional random fields
• Sparse plus low-rank autoregressive identification in neuroimaging time series
• A Junction Tree Framework for Undirected Graphical Model Selection
• High-Dimensional Covariance Decomposition into Sparse Markov and Independence Domains
• Estimation for ultra-high dimensional factor model: a pivotal variable detection based approach
• Learning latent causal graphs via mixture oracles
• Nonsparse learning with latent variables
• Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation
• Factorial graphical lasso for dynamic networks
• Partial Gaussian Graphical Model Estimation
• General parameterized proximal point algorithm with applications in statistical learning
• Dual Smoothing and Level Set Techniques for Variational Matrix Decomposition
• Learning the effect of latent variables in Gaussian Graphical models with unobserved variables
• Learning Restricted Boltzmann Machines with Sparse Latent Variables
• Efficient Covariance Estimation from Temporal Data
• Nonparametric mixture of Gaussian graphical models
• Learning Exponential Family Graphical Models with Latent Variables using Regularized Conditional Likelihood
• Robust Sparse Reduced Rank Regression in High Dimensions
• Robust Identification of "Sparse Plus Low-rank" Graphical Models: An Optimization Approach
• Bi-cross-validation for factor analysis
• Fast Algorithms for Learning Latent Variables in Graphical Models
• Learning a Common Substructure of Multiple Graphical Gaussian Models
• A Computationally Efficient Method for Learning Exponential Family Distributions
• Adaptive Reference-Guided Estimation of Principal Component Subspace in High Dimensions
• Endogenous Representation of Asset Returns
• A Sparsity Inducing Nuclear-Norm Estimator (SpINNEr) for Matrix-Variate Regression in Brain Connectivity Analysis
• Semiblind subgraph reconstruction in Gaussian graphical models
• Fast and Accurate Graph Learning for Huge Data via Minipatch Ensembles
• Group induced graphical lasso allows for discovery of molecular pathways-pathways interactions
• California Reservoir Drought Sensitivity and Exhaustion Risk Using Statistical Graphical Models
• A dual-primal balanced augmented Lagrangian method for linearly constrained convex programming
• Modeling Website Visits
• Identifying Nonlinear 1-Step Causal Influences in Presence of Latent Variables
• Factor Analysis on Citation, Using a Combined Latent and Logistic Regression Model
• From Boltzmann Machines to Neural Networks and Back Again
• High-dimensional covariance matrix estimation using a low-rank and diagonal decomposition
• Single Index Latent Variable Models for Network Topology Inference
• Exponential Family Graphical Models: Correlated Replicates and Unmeasured Confounders, with Applications to fMRI Data
• Estimation of Graphical Models through Structured Norm Minimization
• A Scalable and Extensible Framework for Superposition-Structured Models
• Learning Semidefinite Regularizers
• Group Symmetry and Covariance Regularization
• Convergence analysis of a relaxed inertial alternating minimization algorithm with applications
• Convex Latent Effect Logit Model via Sparse and Low-rank Decomposition
• Study of Compressed Randomized UTV Decompositions for Low-Rank Matrix Approximations in Data Science
• Topology Inference over Networks with Nonlinear Coupling