Adaptive, Distribution-Free Prediction Intervals for Deep Networks

Abstract

The machine learning literature contains several constructions for prediction intervals that are intuitively reasonable but ultimately ad-hoc in that they do not come with provable performance guarantees. We present methods from the statistics literature that can be used efficiently with neural networks under minimal assumptions with guaranteed performance. We propose a neural network that outputs three values instead of a single point estimate and optimizes a loss function motivated by the standard quantile regression loss. We provide two prediction interval methods with finite sample coverage guarantees solely under the assumption that the observations are independent and identically distributed. The first method leverages the conformal inference framework and provides average coverage. The second method provides a new, stronger guarantee by conditioning on the observed data. Lastly, our loss function does not compromise the predictive accuracy of the network like other prediction interval methods. We demonstrate the ease of use of our procedures as well as its improvements over other methods on both simulated and real data. As most deep networks can easily be modified by our method to output predictions with valid prediction intervals, its use should become standard practice, much like reporting standard errors along with mean estimates.

Cited by in corpus (20)

• Testing for Outliers with Conformal p-values
• Nested conformal prediction and quantile out-of-bag ensemble methods
• Adaptive Conformal Inference Under Distribution Shift
• Conformal prediction with localization
• DeepGLEAM: A hybrid mechanistic and deep learning model for COVID-19 forecasting
• PIVEN: A Deep Neural Network for Prediction Intervals with Specific Value Prediction
• Conformal Prediction using Conditional Histograms
• Selection and Aggregation of Conformal Prediction Sets
• Exchangeability, Conformal Prediction, and Rank Tests
• With Malice Towards None: Assessing Uncertainty via Equalized Coverage
• Improving Conditional Coverage via Orthogonal Quantile Regression
• Calibrated Multiple-Output Quantile Regression with Representation Learning
• Few-shot Conformal Prediction with Auxiliary Tasks
• Learning Risk-aware Costmaps for Traversability in Challenging Environments
• Understanding the Under-Coverage Bias in Uncertainty Estimation
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission
• AutoCP: Automated Pipelines for Accurate Prediction Intervals
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting
• Conformal Prediction Intervals for Neural Networks Using Cross Validation
• Synthesizing Machine Learning Programs with PAC Guarantees via Statistical Sketching