To Trust Or Not To Trust A Classifier

Abstract

Knowing when a classifier's prediction can be trusted is useful in many applications and critical for safely using AI. While the bulk of the effort in machine learning research has been towards improving classifier performance, understanding when a classifier's predictions should and should not be trusted has received far less attention. The standard approach is to use the classifier's discriminant or confidence score; however, we show there exists an alternative that is more effective in many situations. We propose a new score, called the trust score, which measures the agreement between the classifier and a modified nearest-neighbor classifier on the testing example. We show empirically that high (low) trust scores produce surprisingly high precision at identifying correctly (incorrectly) classified examples, consistently outperforming the classifier's confidence score as well as many other baselines. Further, under some mild distributional assumptions, we show that if the trust score for an example is high (low), the classifier will likely agree (disagree) with the Bayes-optimal classifier. Our guarantees consist of non-asymptotic rates of statistical consistency under various nonparametric settings and build on recent developments in topological data analysis.

NIPS 2018

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• On Calibration of Modern Neural Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Online Learning with Abstention
• Density Level Set Estimation on Manifolds with DBSCAN
• Adaptive Hausdorff estimation of density level sets

Cited by in corpus (50)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
• Correction of AI systems by linear discriminants: Probabilistic foundations
• BERT Loses Patience: Fast and Robust Inference with Early Exit
• Deep Weighted Averaging Classifiers
• A New Approach for Interpretability and Reliability in Clinical Risk Prediction: Acute Coronary Syndrome Scenario
• Uncertainty Sets for Image Classifiers using Conformal Prediction
• Outside the Box: Abstraction-Based Monitoring of Neural Networks
• What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
• Consistent Estimators for Learning to Defer to an Expert
• HashTran-DNN: A Framework for Enhancing Robustness of Deep Neural Networks against Adversarial Malware Samples
• How Much Can We Really Trust You? Towards Simple, Interpretable Trust Quantification Metrics for Deep Neural Networks
• ProtoAttend: Attention-Based Prototypical Learning
• FUTURE-AI: Guiding Principles and Consensus Recommendations for Trustworthy Artificial Intelligence in Medical Imaging
• Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation
• Anomalous Example Detection in Deep Learning: A Survey
• Local intrinsic dimensionality estimators based on concentration of measure
• Quizbowl: The Case for Incremental Question Answering
• Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples
• To Trust or Not to Trust a Regressor: Estimating and Explaining Trustworthiness of Regression Predictions
• Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
• A Trustworthiness Score to Evaluate DNN Predictions
• Uncertainty Prediction for Deep Sequential Regression Using Meta Models
• Towards Principled Uncertainty Estimation for Deep Neural Networks
• Justification-Based Reliability in Machine Learning
• Insights into Fairness through Trust: Multi-scale Trust Quantification for Financial Deep Learning
• Few-shot Conformal Prediction with Auxiliary Tasks
• Toward Metrics for Differentiating Out-of-Distribution Sets
• Adaptive Deep Learning for Entity Resolution by Risk Analysis
• Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data
• Locally Adaptive Label Smoothing for Predictive Churn
• Online Anomaly Detection in Surveillance Videos with Asymptotic Bounds on False Alarm Rate
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission
• Improving Explainable Object-induced Model through Uncertainty for Automated Vehicles
• Uncertainty Propagation in Node Classification
• Test Sample Accuracy Scales with Training Sample Density in Neural Networks
• Trinary Tools for Continuously Valued Binary Classifiers
• The Compact Support Neural Network
• Conformance Constraint Discovery: Measuring Trust in Data-Driven Systems
• A Survey on Assessing the Generalization Envelope of Deep Neural Networks: Predictive Uncertainty, Out-of-distribution and Adversarial Samples
• MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator
• On Deep Neural Network Calibration by Regularization and its Impact on Refinement
• Test for non-negligible adverse shifts
• BED: Bi-Encoder-Based Detectors for Out-of-Distribution Detection
• Toward Auto-evaluation with Confidence-based Category Relation-aware Regression
• Logit-based Uncertainty Measure in Classification