Deeper Connections between Neural Networks and Gaussian Processes Speed-up Active Learning

Abstract

Active learning methods for neural networks are usually based on greedy criteria which ultimately give a single new design point for the evaluation. Such an approach requires either some heuristics to sample a batch of design points at one active learning iteration, or retraining the neural network after adding each data point, which is computationally inefficient. Moreover, uncertainty estimates for neural networks sometimes are overconfident for the points lying far from the training sample. In this work we propose to approximate Bayesian neural networks (BNN) by Gaussian processes, which allows us to update the uncertainty estimates of predictions efficiently without retraining the neural network, while avoiding overconfident uncertainty prediction for out-of-sample points. In a series of experiments on real-world data including large-scale problems of chemical and physical modeling, we show superiority of the proposed approach over the state-of-the-art methods.

References in corpus (10)

• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• Deep Bayesian Active Learning with Image Data
• Deep Neural Networks as Gaussian Processes
• Gaussian Processes for Big Data
• Variational Bayesian Inference with Stochastic Search
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Dropout-based Active Learning for Regression
• Differentiable Compositional Kernel Learning for Gaussian Processes
• Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles

Cited by in corpus (3)

• Accelerating high-throughput virtual screening through molecular pool-based active learning
• Finite Versus Infinite Neural Networks: an Empirical Study
• Bayesian active learning for production, a systematic study and a reusable library