Fast and reliable uncertainty quantification with neural network ensembles for industrial image classification

Abstract

Image classification with neural networks (NNs) is widely used in industrial processes, situations where the model likely encounters unknown objects during deployment, i.e., out-of-distribution (OOD) data. Worryingly, NNs tend to make confident yet incorrect predictions when confronted with OOD data. To increase the models' reliability, they should quantify the uncertainty in their own predictions, communicating when the output should (not) be trusted. Deep ensembles, composed of multiple independent NNs, have been shown to perform strongly but are computationally expensive. Recent research has proposed more efficient NN ensembles, namely the snapshot, batch, and multi-input multi-output ensemble. This study investigates the predictive and uncertainty performance of efficient NN ensembles in the context of image classification for industrial processes. It is the first to provide a comprehensive comparison and it proposes a novel Diversity Quality metric to quantify the ensembles' performance on the in-distribution and OOD sets in one single metric. The results highlight the batch ensemble as a cost-effective and competitive alternative to the deep ensemble. It matches the deep ensemble in both uncertainty and accuracy while exhibiting considerable savings in training time, test time, and memory storage.

Accepted Manuscript version of an article published in Annals of Operations Research

References in corpus (10)

• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Deep Ensembles: A Loss Landscape Perspective
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
• Explainability through uncertainty: Trustworthy decision-making with neural networks
• Towards Sim-to-Real Industrial Parts Classification with Synthetic Dataset
• A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks
• Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning