Improving Output Uncertainty Estimation and Generalization in Deep Learning via Neural Network Gaussian Processes

Abstract

We propose a simple method that combines neural networks and Gaussian processes. The proposed method can estimate the uncertainty of outputs and flexibly adjust target functions where training data exist, which are advantages of Gaussian processes. The proposed method can also achieve high generalization performance for unseen input configurations, which is an advantage of neural networks. With the proposed method, neural networks are used for the mean functions of Gaussian processes. We present a scalable stochastic inference procedure, where sparse Gaussian processes are inferred by stochastic variational inference, and the parameters of neural networks and kernels are estimated by stochastic gradient descent methods, simultaneously. We use two real-world spatio-temporal data sets to demonstrate experimentally that the proposed method achieves better uncertainty estimation and generalization performance than neural networks and Gaussian processes.

Cited by in corpus (12)

• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• Meta-Learning Mean Functions for Gaussian Processes
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• What do you Mean? The Role of the Mean Function in Bayesian Optimisation
• Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel
• Variational Implicit Processes
• Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series
• Calibrated simplex-mapping classification
• Deep Mixed Effect Model using Gaussian Processes: A Personalized and Reliable Prediction for Healthcare
• Efficient Transfer Bayesian Optimization with Auxiliary Information
• Few-shot Learning for Spatial Regression
• A Novel Regression Loss for Non-Parametric Uncertainty Optimization