PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction

Abstract

We propose an algorithm combining calibrated prediction and generalization bounds from learning theory to construct confidence sets for deep neural networks with PAC guarantees---i.e., the confidence set for a given input contains the true label with high probability. We demonstrate how our approach can be used to construct PAC confidence sets on ResNet for ImageNet, a visual object tracking model, and a dynamics model for the half-cheetah reinforcement learning problem.

arXiv commentAccepted to ICLR 2020

References in corpus (5)

• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Calibrated Model-Based Deep Reinforcement Learning
• Leave-one-out prediction intervals in linear regression models with many variables
• Learning Confidence Sets using Support Vector Machines

Cited by in corpus (11)

• Testing for Outliers with Conformal p-values
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
• Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation
• Private Prediction Sets
• PAC Confidence Predictions for Deep Neural Network Classifiers
• PAC Prediction Sets Under Covariate Shift
• Reachability of Linear Uncertain Systems: Sampling Based Approaches
• Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling
• Synthesizing Machine Learning Programs with PAC Guarantees via Statistical Sketching
• Learning Models for Actionable Recourse
• MPC-guided Imitation Learning of Neural Network Policies for the Artificial Pancreas