Deep Gaussian Processes for Regression using Approximate Expectation Propagation

Abstract

Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.

References in corpus (3)

• Variational Auto-encoded Deep Gaussian Processes
• Nested Variational Compression in Deep Gaussian Processes
• Stochastic Expectation Propagation

Cited by in corpus (52)

• Deep Neural Networks as Gaussian Processes
• Variational Continual Learning
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• Random Feature Expansions for Deep Gaussian Processes
• Bayesian Layers: A Module for Neural Network Uncertainty
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Physics-informed deep generative models
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning
• Hydra: Preserving Ensemble Diversity for Model Distillation
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Improving and Understanding Variational Continual Learning
• An Anomaly Detection Method for Satellites Using Monte Carlo Dropout
• On the Importance of Strong Baselines in Bayesian Deep Learning
• Streaming Sparse Gaussian Process Approximations
• Deep Gaussian Processes with Convolutional Kernels
• Multi-class Gaussian Process Classification with Noisy Inputs
• Subspace Inference for Bayesian Deep Learning
• Deep Gaussian Process-Based Bayesian Inference for Contaminant Source Localization
• The Gaussian Process Autoregressive Regression Model (GPAR)
• Scalable Exact Inference in Multi-Output Gaussian Processes
• Bayesian Optimization using Deep Gaussian Processes
• Benchmarking the Neural Linear Model for Regression
• Inference in Deep Networks in High Dimensions
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• Measuring Uncertainty in Signal Fingerprinting with Gaussian Processes Going Deep
• Variational Implicit Processes
• Robust Deep Gaussian Processes
• Radial and Directional Posteriors for Bayesian Neural Networks
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• Why bigger is not always better: on finite and infinite neural networks
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Sparse Spectrum Warped Input Measures for Nonstationary Kernel Learning
• Bayesian active learning for choice models with deep Gaussian processes
• SDG-L: A Semiparametric Deep Gaussian Process based Framework for Battery Capacity Prediction
• Energy-Based Processes for Exchangeable Data
• Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the Predictive Uncertainties
• On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes
• Inter-domain Deep Gaussian Processes
• Transport Gaussian Processes for Regression
• Forecasting Wireless Demand with Extreme Values using Feature Embedding in Gaussian Processes
• The Renyi Gaussian Process: Towards Improved Generalization
• Convolutional Normalizing Flows for Deep Gaussian Processes
• Efficient Deep Gaussian Process Models for Variable-Sized Input
• Warped Input Gaussian Processes for Time Series Forecasting
• Epoch-evolving Gaussian Process Guided Learning
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks