High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables

Abstract

In this work we address the problem of approximating high-dimensional data with a low-dimensional representation. We make the following contributions. We propose an inverse regression method which exchanges the roles of input and response, such that the low-dimensional variable becomes the regressor, and which is tractable. We introduce a mixture of locally-linear probabilistic mapping model that starts with estimating the parameters of inverse regression, and follows with inferring closed-form solutions for the forward parameters of the high-dimensional regression problem of interest. Moreover, we introduce a partially-latent paradigm, such that the vector-valued response variable is composed of both observed and latent entries, thus being able to deal with data contaminated by experimental artifacts that cannot be explained with noise models. The proposed probabilistic formulation could be viewed as a latent-variable augmentation of regression. We devise expectation-maximization (EM) procedures based on a data augmentation strategy which facilitates the maximum-likelihood search over the model parameters. We propose two augmentation schemes and we describe in detail the associated EM inference procedures that may well be viewed as generalizations of a number of EM regression, dimension reduction, and factor analysis algorithms. The proposed framework is validated with both synthetic and real data. We provide experimental evidence that our method outperforms several existing regression techniques.

References in corpus (2)

• Fisher Lecture: Dimension Reduction in Regression
• Residual Component Analysis: Generalising PCA for more flexible inference in linear-Gaussian models

Cited by in corpus (15)

• Robust Head-Pose Estimation Based on Partially-Latent Mixture of Linear Regressions
• Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regression
• Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models
• Plane-extraction from depth-data using a Gaussian mixture regression model
• Hyper-Spectral Image Analysis with Partially-Latent Regression and Spatial Markov Dependencies
• Prediction regions through Inverse Regression
• The impact of asteroid shapes and topographies on their reflectance spectroscopy
• Estimation, Confidence Intervals, and Large-Scale Hypotheses Testing for High-Dimensional Mixed Linear Regression
• Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers
• An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling
• Regression-based heterogeneity analysis to identify overlapping subgroup structure in high-dimensional data
• Approximation results regarding the multiple-output mixture of linear experts model
• A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models
• Simulation-based inference for stochastic nonlinear mixed-effects models with applications in systems biology
• Relative Transfer Function Inverse Regression from Low Dimensional Manifold