Epistemic Neural Networks

Abstract

Intelligence relies on an agent's knowledge of what it does not know. This capability can be assessed based on the quality of joint predictions of labels across multiple inputs. In principle, ensemble-based approaches produce effective joint predictions, but the computational costs of training large ensembles can become prohibitive. We introduce the epinet: an architecture that can supplement any conventional neural network, including large pretrained models, and can be trained with modest incremental computation to estimate uncertainty. With an epinet, conventional neural networks outperform very large ensembles, consisting of hundreds or more particles, with orders of magnitude less computation. The epinet does not fit the traditional framework of Bayesian neural networks. To accommodate development of approaches beyond BNNs, such as the epinet, we introduce the epistemic neural network (ENN) as an interface for models that produce joint predictions.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Neural Networks as Gaussian Processes
• What Are Bayesian Neural Network Posteriors Really Like?
• Bootstrapped Thompson Sampling and Deep Exploration
• Variational Gaussian Dropout is not Bayesian
• Hypermodels for Exploration

Cited by in corpus (1)

• Monotone-Value Neural Networks: Exploiting Preference Monotonicity in Combinatorial Assignment