Controlled abstention neural networks for identifying skillful predictions for regression problems

Abstract

The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed "forecasts of opportunity". When these opportunities are not present, scientists need prediction systems that are capable of saying "I don't know." We introduce a novel loss function, termed "abstention loss", that allows neural networks to identify forecasts of opportunity for regression problems. The abstention loss works by incorporating uncertainty in the network's prediction to identify the more confident samples and abstain (say "I don't know") on the less confident samples. The abstention loss is designed to determine the optimal abstention fraction, or abstain on a user-defined fraction via a PID controller. Unlike many methods for attaching uncertainty to neural network predictions post-training, the abstention loss is applied during training to preferentially learn from the more confident samples. The abstention loss is built upon a standard computer science method. While the standard approach is itself a simple yet powerful tool for incorporating uncertainty in regression problems, we demonstrate that the abstention loss outperforms this more standard method for the synthetic climate use cases explored here. The implementation of proposed loss function is straightforward in most network architectures designed for regression, as it only requires modification of the output layer and loss function.

submitted to the Journal of Advances of Earth System Modeling

References in corpus (7)

• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• Indicator patterns of forced change learned by an artificial neural network
• SelectiveNet: A Deep Neural Network with an Integrated Reject Option
• Controlled abstention neural networks for identifying skillful predictions for classification problems
• Identifying Opportunities for Skillful Weather Prediction with Interpretable Neural Networks
• Adding Uncertainty to Neural Network Regression Tasks in the Geosciences
• A Unified Plug-and-Play Framework for Effective Data Denoising and Robust Abstention

Cited by in corpus (6)

• Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation for Earth System Science Applications
• Evaluation of Machine Learning Techniques for Forecast Uncertainty Quantification
• Controlled abstention neural networks for identifying skillful predictions for classification problems
• CIRA Guide to Custom Loss Functions for Neural Networks in Environmental Sciences -- Version 1
• Predicting Tropical Cyclone Track Forecast Errors using a Probabilistic Neural Network
• Probabilistic neural networks for predicting energy dissipation rates in geophysical turbulent flows