GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

Abstract

Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call. BBMM reduces the asymptotic complexity of exact GP inference from $O(n^3)$ to $O(n^2)$. Adapting this algorithm to scalable approximations and complex GP models simply requires a routine for efficient matrix-matrix multiplication with the kernel and its derivative. In addition, BBMM uses a specialized preconditioner to substantially speed up convergence. In experiments we show that BBMM effectively uses GPU hardware to dramatically accelerate both exact GP inference and scalable approximations. Additionally, we provide GPyTorch, a software platform for scalable GP inference via BBMM, built on PyTorch.

NeurIPS 2018. Most recent version includes additional details on preconditioned BBMM

References in corpus (10)

• Auto-Encoding Variational Bayes
• Caffe: Convolutional Architecture for Fast Feature Embedding
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Thoughts on Massively Scalable Gaussian Processes
• Product Kernel Interpolation for Scalable Gaussian Processes
• Scalable Log Determinants for Gaussian Process Kernel Learning
• Constant-Time Predictive Distributions for Gaussian Processes
• Improved stochastic trace estimation using mutually unbiased bases

Cited by in corpus (116)

• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• Kernel Operations on the GPU, with Autodiff, without Memory Overflows
• Bayesian Layers: A Module for Neural Network Uncertainty
• Residual Correlation in Graph Neural Network Regression
• Multi-Fidelity Cost-Aware Bayesian Optimization
• On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty
• Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domains
• On long-term fatigue damage estimation for a floating offshore wind turbine using a surrogate model
• ET-AL: Entropy-Targeted Active Learning for Bias Mitigation in Materials Data
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Exact Gaussian Processes on a Million Data Points
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• A Fast, 2D Gaussian Process Method Based on Celerite: Applications to Transiting Exoplanet Discovery and Characterization
• Probabilistic Deep Learning to Quantify Uncertainty in Air Quality Forecasting
• Meta-Learning Mean Functions for Gaussian Processes
• BNNpriors: A library for Bayesian neural network inference with different prior distributions
• Machine Learning with bond information for local structure optimizations in surface science
• Incorporating structured assumptions with probabilistic graphical models in fMRI data analysis
• Scalable Constrained Bayesian Optimization
• Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting
• Deep Factors with Gaussian Processes for Forecasting
• Model Predictive Robustness of Signal Temporal Logic Predicates
• Quantifying out-of-station waiting time in oversaturated urban metro systems
• Numerical issues in maximum likelihood parameter estimation for Gaussian process interpolation
• Sparse Gaussian processes for solving nonlinear PDEs
• On Controller Tuning with Time-Varying Bayesian Optimization
• Non-parametric reconstruction of cosmological observables using Gaussian Processes Regression
• Adversarial Attack for Uncertainty Estimation: Identifying Critical Regions in Neural Networks
• Convergence of Sparse Variational Inference in Gaussian Processes Regression
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically
• Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces
• Probabilistic Metamodels for an Efficient Characterization of Complex Driving Scenarios
• Splitting Gaussian Process Regression for Streaming Data
• Improving predictions of Bayesian neural nets via local linearization
• Learning battery model parameter dynamics from data with recursive Gaussian process regression
• A Global-Local Approximation Framework for Large-Scale Gaussian Process Modeling
• Scaling Gaussian Processes with Derivative Information Using Variational Inference
• Transforming Gaussian Processes With Normalizing Flows
• Bayesian Optimization with High-Dimensional Outputs
• Parametric Gaussian Process Regressors
• Learning and Interpreting Multi-Multi-Instance Learning Networks
• Short-Term Solar Irradiance Forecasting Using Calibrated Probabilistic Models
• How Bayesian Should Bayesian Optimisation Be?
• Probabilistic selection of inducing points in sparse Gaussian processes
• Safe Guaranteed Exploration for Non-linear Systems
• A High-Performance Accelerator for Super-Resolution Processing on Embedded GPU
• Distribution Regression for Sequential Data
• CorrAttack: Black-box Adversarial Attack with Structured Search
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• A Principled Approach to Design Using High Fidelity Fluid-Structure Interaction Simulations
• Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces
• Molecular Energy Learning Using Alternative Blackbox Matrix-Matrix Multiplication Algorithm for Exact Gaussian Process
• Multipoint-BAX: A New Approach for Efficiently Tuning Particle Accelerator Emittance via Virtual Objectives
• Rapidly Personalizing Mobile Health Treatment Policies with Limited Data
• Bayesian Optimization Algorithms for Accelerator Physics
• Variational Bayesian surrogate modelling with application to robust design optimisation
• Bayesian Active Search on Parameter Space: a 95 GeV Spin-0 Resonance in the ($B-L$)SSM
• Spatio-Temporal Variational Gaussian Processes
• Active Learning with Gaussian Processes for High Throughput Phenotyping
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Pathwise Conditioning of Gaussian Processes
• Multi-Sparse Gaussian Process: Learning based Semi-Parametric Control
• Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Models
• Bayesian Optimization for Ion Beam Centroid Correction
• MOLPIPx: an end-to-end differentiable package for permutationally invariant polynomials in Python and Rust
• Uncertainty-Aware Time-to-Event Prediction using Deep Kernel Accelerated Failure Time Models
• Deep Curvature Suite
• Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes
• Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees
• Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
• Fast Kronecker Matrix-Matrix Multiplication on GPUs
• Gaussian Process Uniform Error Bounds with Unknown Hyperparameters for Safety-Critical Applications
• Rectangular Flows for Manifold Learning
• Fast and Accurate Gaussian Kernel Ridge Regression Using Matrix Decompositions for Preconditioning
• Deep Gaussian Processes for Air Quality Inference
• Kernel Conditional Density Operators
• Bias-Free Scalable Gaussian Processes via Randomized Truncations
• Impact of redshift distribution uncertainties on Lyman-break galaxy cosmological parameter inference
• Path Imputation Strategies for Signature Models of Irregular Time Series
• Neural Likelihoods for Multi-Output Gaussian Processes
• High-Dimensional Gaussian Process Inference with Derivatives
• High-Dimensional Bayesian Optimization via Nested Riemannian Manifolds
• Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features
• Quantifying Predictive Uncertainty in Medical Image Analysis with Deep Kernel Learning
• Deep Kernel Learning for Mortality Prediction in the Face of Temporal Shift
• Kernel-Matrix Determinant Estimates from stopped Cholesky Decomposition
• MBORE: Multi-objective Bayesian Optimisation by Density-Ratio Estimation
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling
• Deconditional Downscaling with Gaussian Processes
• DRF: A Framework for High-Accuracy Autonomous Driving Vehicle Modeling
• Bayesian Probabilistic Numerical Integration with Tree-Based Models
• Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients
• The Renyi Gaussian Process: Towards Improved Generalization
• A Hierarchical Approach to Scaling Batch Active Search Over Structured Data
• Scalable Gaussian Process Regression for Kernels with a Non-Stationary Phase
• Risk factor identification for incident heart failure using neural network distillation and variable selection
• Planning from Images with Deep Latent Gaussian Process Dynamics
• Forecasting Emergency Department Capacity Constraints for COVID Isolation Beds
• Bayesian Meta-Learning Through Variational Gaussian Processes
• Optimizing High-Dimensional Physics Simulations via Composite Bayesian Optimization
• What classifiers know what they don't?
• Deep Bayesian Estimation for Dynamic Treatment Regimes with a Long Follow-up Time
• Barely Biased Learning for Gaussian Process Regression
• Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations
• Applications of Conjugate Gradient in Bayesian computation
• Balanced Depth Completion between Dense Depth Inference and Sparse Range Measurements via KISS-GP
• Achieving Diversity in Objective Space for Sample-efficient Search of Multiobjective Optimization Problems
• Personalizing Intervention Probabilities By Pooling
• Bayesian Variable Selection for Single Index Logistic Model
• Streamlined Empirical Bayes Fitting of Linear Mixed Models in Mobile Health
• PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Python
• Collaborative Multidisciplinary Design Optimization with Neural Networks