Kernel dimension reduction in regression

Abstract

We present a new methodology for sufficient dimension reduction (SDR). Our methodology derives directly from the formulation of SDR in terms of the conditional independence of the covariate $X$ from the response $Y$, given the projection of $X$ on the central subspace [cf. J. Amer. Statist. Assoc. 86 (1991) 316--342 and Regression Graphics (1998) Wiley]. We show that this conditional independence assertion can be characterized in terms of conditional covariance operators on reproducing kernel Hilbert spaces and we show how this characterization leads to an $M$-estimator for the central subspace. The resulting estimator is shown to be consistent under weak conditions; in particular, we do not have to impose linearity or ellipticity conditions of the kinds that are generally invoked for SDR methods. We also present empirical results showing that the new methodology is competitive in practice.

Published in at http://dx.doi.org/10.1214/08-AOS637 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (1)

• Contour regression: A general approach to dimension reduction

Cited by in corpus (46)

• Kernel Mean Embedding of Distributions: A Review and Beyond
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• Linear Dimensionality Reduction: Survey, Insights, and Generalizations
• Principal support vector machines for linear and nonlinear sufficient dimension reduction
• A general theory for nonlinear sufficient dimension reduction: Formulation and estimation
• Sufficient dimension reduction based on an ensemble of minimum average variance estimators
• Kernel Feature Selection via Conditional Covariance Minimization
• Adaptive Kernel Kalman Filter
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Characteristic and Universal Tensor Product Kernels
• An adaptive composite quantile approach to dimension reduction
• A Rigorous Theory of Conditional Mean Embeddings
• Cross-Domain Object Matching with Model Selection
• Estimating sufficient reductions of the predictors in abundant high-dimensional regressions
• Discussion of: Brownian distance covariance
• Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nyström Method, and Use of Kernels in Machine Learning: Tutorial and Survey
• Kernel Dependence Regularizers and Gaussian Processes with Applications to Algorithmic Fairness
• Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence
• Conditional mean embeddings as regressors - supplementary
• Nonlinear dimension reduction for surrogate modeling using gradient information
• Sparse Linear Regression With Missing Data
• Solving Interpretable Kernel Dimension Reduction
• Subspace Perspective on Canonical Correlation Analysis: Dimension Reduction and Minimax Rates
• Direct Estimation of the Derivative of Quadratic Mutual Information with Application in Supervised Dimension Reduction
• Deep Dimension Reduction for Supervised Representation Learning
• Learning nonlinear level sets for dimensionality reduction in function approximation
• SERAPH: Semi-supervised Metric Learning Paradigm with Hyper Sparsity
• Comment on "Support Vector Machines with Applications"
• Tail inverse regression for dimension reduction with extreme response
• Learning Integral Representations of Gaussian Processes
• A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
• Ultrahigh Dimensional Feature Selection via Kernel Canonical Correlation Analysis
• Sufficient Component Analysis for Supervised Dimension Reduction
• Filtering with State-Observation Examples via Kernel Monte Carlo Filter
• Gradient-based kernel dimension reduction for supervised learning
• Conditional Bures Metric for Domain Adaptation
• An RKHS-Based Semiparametric Approach to Nonlinear Sufficient Dimension Reduction
• Policy Search with High-Dimensional Context Variables
• A U-statistic Approach to Hypothesis Testing for Structure Discovery in Undirected Graphical Models
• Conditional Density Estimation with Dimensionality Reduction via Squared-Loss Conditional Entropy Minimization
• Sobolev Norm Learning Rates for Conditional Mean Embeddings
• Spectral Non-Convex Optimization for Dimension Reduction with Hilbert-Schmidt Independence Criterion
• Sufficient Dimension Reduction for High-Dimensional Regression and Low-Dimensional Embedding: Tutorial and Survey
• Dependence versus Conditional Dependence in Local Causal Discovery from Gene Expression Data
• Learning sparse gradients for variable selection and dimension reduction
• Bridging linearity-based and kernel-based sufficient dimension reduction