Kernel Mean Embedding of Distributions: A Review and Beyond
arXiv:1605.09522 · doi:10.1561/2200000060
Abstract
A Hilbert space embedding of a distribution---in short, a kernel mean embedding---has recently emerged as a powerful tool for machine learning and inference. The basic idea behind this framework is to map distributions into a reproducing kernel Hilbert space (RKHS) in which the whole arsenal of kernel methods can be extended to probability measures. It can be viewed as a generalization of the original "feature map" common to support vector machines (SVMs) and other kernel methods. While initially closely associated with the latter, it has meanwhile found application in fields ranging from kernel machines and probabilistic modeling to statistical inference, causal discovery, and deep learning. The goal of this survey is to give a comprehensive review of existing work and recent advances in this research area, and to discuss the most challenging issues and open problems that could lead to new research directions. The survey begins with a brief introduction to the RKHS and positive definite kernels which forms the backbone of this survey, followed by a thorough discussion of the Hilbert space embedding of marginal distributions, theoretical guarantees, and a review of its applications. The embedding of distributions enables us to apply RKHS methods to probability measures which prompts a wide range of applications such as kernel two-sample testing, independent testing, and learning on distributional data. Next, we discuss the Hilbert space embedding for conditional distributions, give theoretical insights, and review some applications. The conditional mean embedding enables us to perform sum, product, and Bayes' rules---which are ubiquitous in graphical model, probabilistic inference, and reinforcement learning---in a non-parametric way. We then discuss relationships between this framework and other related areas. Lastly, we give some suggestions on future research directions.
arXiv comment147 pages; this is the final version
References in corpus (52)
- Learning Transferable Features with Deep Adaptation Networks
- Spectral Networks and Locally Connected Networks on Graphs
- Unsupervised Domain Adaptation with Residual Transfer Networks
- Brownian distance covariance
- Discrete Signal Processing on Graphs: Sampling Theory
- Equivalence of distance-based and RKHS-based statistics in hypothesis testing
- Domain Generalization via Invariant Feature Representation
- Generative Moment Matching Networks
- Kernel-based Conditional Independence Test and Application in Causal Discovery
- UAV-Assisted Heterogeneous Networks for Capacity Enhancement
- Convolutional Neural Network Architectures for Signals Supported on Graphs
- Distinguishing cause from effect using observational data: methods and benchmarks
- Kernel dimension reduction in regression
- Sampling of graph signals with successive local aggregations
- Stability Properties of Graph Neural Networks
- Training generative neural networks via Maximum Mean Discrepancy optimization
- A Kernel Test of Goodness of Fit
- Learning from Distributions via Support Measure Machines
- Sharp analysis of low-rank kernel matrix approximations
- Density Estimation in Infinite Dimensional Exponential Families
- Kernel Belief Propagation
- Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions
- k-NN Regression Adapts to Local Intrinsic Dimension
- Testing for Homogeneity with Kernel Fisher Discriminant Analysis
- Relational Forward Models for Multi-Agent Learning
- Optimal Rates for Random Fourier Features
- FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Test
- A Test of Relative Similarity For Model Selection in Generative Models
- On the optimal estimation of probability measures in weak and strong topologies
- A Kernel Independence Test for Random Processes
- On the Error of Random Fourier Features
- Kernel Mean Shrinkage Estimators
- Gradient-free Hamiltonian Monte Carlo with Efficient Kernel Exponential Families
- Sequential Kernel Herding: Frank-Wolfe Optimization for Particle Filtering
- Kernel Adaptive Metropolis-Hastings
- Bayesian Quadratic Network Game Filters
- Optimally-Weighted Herding is Bayesian Quadrature
- A Kernel Test for Three-Variable Interactions
- A Wild Bootstrap for Degenerate Kernel Tests
- Herding Dynamic Weights for Partially Observed Random Field Models
- Convergence guarantees for kernel-based quadrature rules in misspecified settings
- Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing
- Hilbert Space Embeddings of POMDPs
- Detecting low-complexity unobserved causes
- Testing Hypotheses by Regularized Maximum Mean Discrepancy
- Kernel Mean Estimation via Spectral Filtering
- Identifying Finite Mixtures of Nonparametric Product Distributions and Causal Inference of Confounders
- Characteristic Kernels and Infinitely Divisible Distributions
- Minimax Estimation of Kernel Mean Embeddings
- A low variance consistent test of relative dependency
- Nonparametric Independence Testing for Small Sample Sizes
- On a link between kernel mean maps and Fraunhofer diffraction, with an application to super-resolution beyond the diffraction limit
Cited by in corpus (58)
- MMD GAN: Towards Deeper Understanding of Moment Matching Network
- Deep transfer operator learning for partial differential equations under conditional shift
- Eigendecompositions of Transfer Operators in Reproducing Kernel Hilbert Spaces
- Kernel-based approximation of the Koopman generator and Schrödinger operator
- Comparison of Maximum Likelihood and GAN-based training of Real NVPs
- Robust Domain Misinformation Detection via Multi-modal Feature Alignment
- Discovering and Deciphering Relationships Across Disparate Data Modalities
- A kernel-based approach to molecular conformation analysis
- Kernel methods for detecting coherent structures in dynamical data
- Anomaly Detection Based on Isolation Mechanisms: A Survey
- Distribution-based Sketching of Single-Cell Samples
- Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
- Learnware: Small Models Do Big
- Kernel method for persistence diagrams via kernel embedding and weight factor
- Nonequilibrium Statistical Mechanics and Optimal Prediction of Partially-Observed Complex Systems
- Supervised Learning with Quantum Measurements
- Automatic and effective discovery of quantum kernels
- Quantum Mean Embedding of Probability Distributions
- Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
- Transparent Single-Cell Set Classification with Kernel Mean Embeddings
- Change Detection of Markov Kernels with Unknown Pre and Post Change Kernel
- Feature-Based Interpolation and Geodesics in the Latent Spaces of Generative Models
- Singular Value Decomposition of Operators on Reproducing Kernel Hilbert Spaces
- Local Group Invariant Representations via Orbit Embeddings
- Distribution regression model with a Reproducing Kernel Hilbert Space approach
- Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
- Steganographer Identification
- When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights
- Detecting Change Intervals with Isolation Distributional Kernel
- Multimodal Multi-User Surface Recognition with the Kernel Two-Sample Test
- Confidence-based Estimators for Predictive Performance in Model Monitoring
- Data-Driven Observability Analysis for Nonlinear Stochastic Systems
- Estimation of copulas via Maximum Mean Discrepancy
- Diffeological statistical models, the Fisher metric and probabilistic mappings
- Linear quadratic control of nonlinear systems with Koopman operator learning and the Nyström method
- Learning Interpretable Collective Variables for Spreading Processes on Networks
- A Kernel Independence Test for Geographical Language Variation
- Testing and Learning on Distributions with Symmetric Noise Invariance
- Koopman operators with intrinsic observables in rigged reproducing kernel Hilbert spaces
- A Distributional Treatment of Real2Sim2Real for Object-Centric Agent Adaptation in Vision-Driven Deformable Linear Object Manipulation
- Towards a Kernel based Uncertainty Decomposition Framework for Data and Models
- Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach
- Approximating Symmetrized Estimators of Scatter via Balanced Incomplete U-Statistics
- Causal survival embeddings: non-parametric counterfactual inference under censoring
- Approximate Bayesian Computation Based on Maxima Weighted Isolation Kernel Mapping
- Model-Free Change Point Detection for Mixing Processes
- Reversible Gromov-Monge Sampler for Simulation-Based Inference
- Ergodic Trajectory Optimization on Generalized Domains Using Maximum Mean Discrepancy
- Fully symmetric kernel quadrature
- Revisiting RIP guarantees for sketching operators on mixture models
- Instrument Space Selection for Kernel Maximum Moment Restriction
- FastPart: Over-Parameterized Stochastic Gradient Descent for Sparse optimisation on Measures
- Signal reconstruction using determinantal sampling
- Generalization Analysis of Transformers in Distribution Regression
- Hyperparameter Learning for Conditional Kernel Mean Embeddings with Rademacher Complexity Bounds
- Minimum Discrepancy Methods in Uncertainty Quantification
- Nonlinear Distribution Regression for Remote Sensing Applications
- Composable Learning with Sparse Kernel Representations