Structure Discovery in Nonparametric Regression through Compositional Kernel Search

Abstract

Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art. We define a space of kernel structures which are built compositionally by adding and multiplying a small number of base kernels. We present a method for searching over this space of structures which mirrors the scientific discovery process. The learned structures can often decompose functions into interpretable components and enable long-range extrapolation on time-series datasets. Our structure search method outperforms many widely used kernels and kernel combination methods on a variety of prediction tasks.

9 pages, 7 figures, To appear in proceedings of the 2013 International Conference on Machine Learning

References in corpus (3)

• Sum-Product Networks: A New Deep Architecture
• Exploring Large Feature Spaces with Hierarchical Multiple Kernel Learning
• Exploiting compositionality to explore a large space of model structures

Cited by in corpus (71)

• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• Gaussian process regression for forecasting battery state of health
• Machine Learning of Linear Differential Equations using Gaussian Processes
• Bayesian optimization of a free-electron laser
• Multivariate Gaussian and Student$-t$ Process Regression for Multi-output Prediction
• How priors of initial hyperparameters affect Gaussian process regression models
• Data-driven nonparametric Li-ion battery ageing model aiming at learning from real operation data -- Part B: Cycling operation
• Automatic Construction and Natural-Language Description of Nonparametric Regression Models
• Avoiding pathologies in very deep networks
• Data-driven nonparametric Li-ion battery ageing model aiming at learning from real operation data -- Part A: Storage operation
• Variational Auto-encoded Deep Gaussian Processes
• Linear Multiple Low-Rank Kernel Based Stationary Gaussian Processes Regression for Time Series
• Temporal Convolutional Neural Networks for Diagnosis from Lab Tests
• Effective Diversity in Population Based Reinforcement Learning
• Differentiable Compositional Kernel Learning for Gaussian Processes
• Product Kernel Interpolation for Scalable Gaussian Processes
• Compositionally-Warped Gaussian Processes
• Recipes for when Physics Fails: Recovering Robust Learning of Physics Informed Neural Networks
• Gaussian process model of 51-dimensional potential energy surface for protonated imidazole dimer
• Learning stable and predictive structures in kinetic systems: Benefits of a causal approach
• The Random Forest Kernel and other kernels for big data from random partitions
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs
• Time series forecasting with Gaussian Processes needs priors
• Generalized Spectral Kernels
• Bayesian optimization for inverse problems in time-dependent quantum dynamics
• Bayesian Learning of Kernel Embeddings
• A Probabilistic Machine Learning Approach to Scheduling Parallel Loops with Bayesian Optimization
• Towards tailoring non-invasive brain stimulation using real-time fMRI and Bayesian optimization
• The Gaussian Process Autoregressive Regression Model (GPAR)
• Quantum Gaussian process model of potential energy surface for a polyatomic molecule
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Fast Kernel Approximations for Latent Force Models and Convolved Multiple-Output Gaussian processes
• Gaussian Process Nowcasting: Application to COVID-19 Mortality Reporting
• GPatt: Fast Multidimensional Pattern Extrapolation with Gaussian Processes
• client2vec: Towards Systematic Baselines for Banking Applications
• Variational Implicit Processes
• A metric for characterizing the arm nonuse workspace in poststroke individuals using a robot arm
• Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
• Time Series Structure Discovery via Probabilistic Program Synthesis
• Divide, Conquer, and Combine: a New Inference Strategy for Probabilistic Programs with Stochastic Support
• Using Regression Kernels to Forecast A Failure to Appear in Court
• Expressive Mortality Models through Gaussian Process Kernels
• Learning to learn generative programs with Memoised Wake-Sleep
• Probabilistic Programming with Gaussian Process Memoization
• Safe Exploration in Markov Decision Processes with Time-Variant Safety using Spatio-Temporal Gaussian Process
• The Automatic Statistician: A Relational Perspective
• Modular Gaussian Processes for Transfer Learning
• Online tuning and light source control using a physics-informed Gaussian process Adi
• Applications of Probabilistic Programming (Master's thesis, 2015)
• Spatiotemporal Prediction of Ambulance Demand using Gaussian Process Regression
• Energy-Based Processes for Exchangeable Data
• Recent Advances in Data-Driven Wireless Communication Using Gaussian Processes: A Comprehensive Survey
• Evolving Gaussian Process kernels from elementary mathematical expressions
• Learning Compositional Sparse Gaussian Processes with a Shrinkage Prior
• Marginalising over Stationary Kernels with Bayesian Quadrature
• Learning GPLVM with arbitrary kernels using the unscented transformation
• Conditional Deep Gaussian Processes: multi-fidelity kernel learning
• Make Hawkes Processes Explainable by Decomposing Self-Triggering Kernels
• Online structural kernel selection for mobile health
• Optimally adaptive Bayesian spectral density estimation for stationary and nonstationary processes
• Kernel Identification Through Transformers
• Model-based Dashboards for Customer Analytics
• Energy Sustainable Mobile Networks via Energy Routing, Learning and Foresighted Optimization
• Warped Gaussian Processes in Remote Sensing Parameter Estimation and Causal Inference
• Interpolation and extrapolation of global potential energy surfaces for polyatomic systems by Gaussian processes with composite kernels
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning
• Hybrid Memoised Wake-Sleep: Approximate Inference at the Discrete-Continuous Interface
• Gaussian Processes with Skewed Laplace Spectral Mixture Kernels for Long-term Forecasting
• A Self-Supervised Framework for Function Learning and Extrapolation
• Nonlinear Matrix Approximation with Radial Basis Function Components