Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences

Abstract

This paper is an attempt to bridge the conceptual gaps between researchers working on the two widely used approaches based on positive definite kernels: Bayesian learning or inference using Gaussian processes on the one side, and frequentist kernel methods based on reproducing kernel Hilbert spaces on the other. It is widely known in machine learning that these two formalisms are closely related; for instance, the estimator of kernel ridge regression is identical to the posterior mean of Gaussian process regression. However, they have been studied and developed almost independently by two essentially separate communities, and this makes it difficult to seamlessly transfer results between them. Our aim is to overcome this potential difficulty. To this end, we review several old and new results and concepts from either side, and juxtapose algorithmic quantities from each framework to highlight close similarities. We also provide discussions on subtle philosophical and theoretical differences between the two approaches.

64 pages

References in corpus (6)

• Brownian distance covariance
• Reproducing kernel Hilbert spaces of Gaussian priors
• On the Equivalence between Herding and Conditional Gradient Algorithms
• Convergence Analysis of Deterministic Kernel-Based Quadrature Rules in Misspecified Settings
• A Bayes-Sard Cubature Method
• Post Selection Inference with Kernels

Cited by in corpus (72)

• Machine Learning in QM/MM Molecular Dynamics Simulations of Condensed-Phase Systems
• Reconstructing QCD Spectral Functions with Gaussian Processes
• Understanding the stochastic partial differential equation approach to smoothing
• Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces
• A Kernel Two-Sample Test for Functional Data
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• Gaussian Process Assisted Active Learning of Physical Laws
• Density estimation with Gaussian processes for gravitational-wave posteriors
• Modelling of physical systems with a Hopf bifurcation using mechanistic models and machine learning
• Uniform Error Bounds for Gaussian Process Regression with Application to Safe Control
• Multi-fidelity data fusion through parameter space reduction with applications to automotive engineering
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• Kernel Dependence Regularizers and Gaussian Processes with Applications to Algorithmic Fairness
• On the Similarity between the Laplace and Neural Tangent Kernels
• Convergence Guarantees for Gaussian Process Means With Misspecified Likelihoods and Smoothness
• KPC: Learning-Based Model Predictive Control with Deterministic Guarantees
• Uniform Error and Posterior Variance Bounds for Gaussian Process Regression with Application to Safe Control
• Efficient force field and energy emulation through partition of permutationally equivalent atoms
• Posterior Variance Analysis of Gaussian Processes with Application to Average Learning Curves
• Region of Attraction for Power Systems using Gaussian Process and Converse Lyapunov Function -- Part I: Theoretical Framework and Off-line Study
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• A Closer Look at Covering Number Bounds for Gaussian Kernels
• Discovery of structure-property relations for molecules via hypothesis-driven active learning over the chemical space
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• On Information Gain and Regret Bounds in Gaussian Process Bandits
• Informative Planning for Worst-Case Error Minimisation in Sparse Gaussian Process Regression
• On the Estimation of Derivatives Using Plug-in Kernel Ridge Regression Estimators
• Information Directed Sampling for Linear Partial Monitoring
• Benign Overfitting and Noisy Features
• Multi-fidelity data fusion for the approximation of scalar functions with low intrinsic dimensionality through active subspaces
• A Domain-Shrinking based Bayesian Optimization Algorithm with Order-Optimal Regret Performance
• Optimal Order Simple Regret for Gaussian Process Bandits
• Uncertainty Estimates for Ordinal Embeddings
• Pathwise Conditioning of Gaussian Processes
• Scalable Thompson Sampling using Sparse Gaussian Process Models
• Learning particle swarming models from data with Gaussian processes
• Scalable Training of Inference Networks for Gaussian-Process Models
• Kriging prediction with isotropic Matérn correlations: Robustness and experimental design
• Sparser Kernel Herding with Pairwise Conditional Gradients without Swap Steps
• SigGPDE: Scaling Sparse Gaussian Processes on Sequential Data
• Statistical Depth Meets Machine Learning: Kernel Mean Embeddings and Depth in Functional Data Analysis
• Convergence of Gaussian process regression: Optimality, robustness, and relationship with kernel ridge regression
• Accumulation of Sub-Sampling Matrices with Applications to Statistical Computation
• No-Regret Bayesian Optimization with Unknown Hyperparameters
• Continuum-Armed Bandits: A Function Space Perspective
• On the optimality of kernels for high-dimensional clustering
• Efficient Batch Black-box Optimization with Deterministic Regret Bounds
• Variance Minimization in the Wasserstein Space for Invariant Causal Prediction
• Smoothness Estimation for Whittle-Matérn Processes on Closed Riemannian Manifolds
• Gaussian Process Uniform Error Bounds with Unknown Hyperparameters for Safety-Critical Applications
• Discrepancy-based Inference for Intractable Generative Models using Quasi-Monte Carlo
• Weighted Gaussian Process Bandits for Non-stationary Environments
• Learning Integral Representations of Gaussian Processes
• Sensing Cox Processes via Posterior Sampling and Positive Bases
• Deep Neural Networks as Point Estimates for Deep Gaussian Processes
• Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits
• Learning curves for Gaussian process regression with power-law priors and targets
• Variational Gaussian Processes: A Functional Analysis View
• Understanding Climate Impacts on Vegetation with Gaussian Processes in Granger Causality
• Fast Statistical Leverage Score Approximation in Kernel Ridge Regression
• The covariance matrix of Green's functions and its application to machine learning
• Modeling Function-Valued Processes with Nonseparable and/or Nonstationary Covariance Structure
• Sobolev Norm Learning Rates for Conditional Mean Embeddings
• Causality and Generalizability: Identifiability and Learning Methods
• Shallow Representation is Deep: Learning Uncertainty-aware and Worst-case Random Feature Dynamics
• Quasi-Bayesian Dual Instrumental Variable Regression
• Accelerated Bayesian Optimization throughWeight-Prior Tuning
• Uniform Generalization Bounds for Overparameterized Neural Networks
• How isotropic kernels perform on simple invariants
• No-Regret Algorithms for Time-Varying Bayesian Optimization
• Bayesian Optimistic Optimisation with Exponentially Decaying Regret
• Rejoinder for "Probabilistic Integration: A Role in Statistical Computation?"