Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions

Abstract

We develop and rigorously evaluate a deep learning based system that can accurately classify skin conditions while detecting rare conditions for which there is not enough data available for training a confident classifier. We frame this task as an out-of-distribution (OOD) detection problem. Our novel approach, hierarchical outlier detection (HOD) assigns multiple abstention classes for each training outlier class and jointly performs a coarse classification of inliers vs. outliers, along with fine-grained classification of the individual classes. We demonstrate the effectiveness of the HOD loss in conjunction with modern representation learning approaches (BiT, SimCLR, MICLe) and explore different ensembling strategies for further improving the results. We perform an extensive subgroup analysis over conditions of varying risk levels and different skin types to investigate how the OOD detection performance changes over each subgroup and demonstrate the gains of our framework in comparison to baselines. Finally, we introduce a cost metric to approximate downstream clinical impact. We use this cost metric to compare the proposed method against a baseline system, thereby making a stronger case for the overall system effectiveness in a real-world deployment scenario.

arXiv commentUnder Review, 19 Pages

References in corpus (17)

• Explaining and Harnessing Adversarial Examples
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• A deep learning system for differential diagnosis of skin diseases
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Deep Ensembles: A Loss Landscape Perspective
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Contrastive Training for Improved Out-of-Distribution Detection
• A Benchmark of Medical Out of Distribution Detection
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Supervised Transfer Learning at Scale for Medical Imaging
• Calibrating Healthcare AI: Towards Reliable and Interpretable Deep Predictive Models
• FRODO: Free rejection of out-of-distribution samples: application to chest x-ray analysis
• Self-Supervised Out-of-Distribution Detection in Brain CT Scans
• Prototypical Clustering Networks for Dermatological Disease Diagnosis
• Out-of-Distribution Detection for Skin Lesion Images with Deep Isolation Forest
• CheXseen: Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays

Cited by in corpus (8)

• Recommendations on test datasets for evaluating AI solutions in pathology
• Learning Optimal Conformal Classifiers
• Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection
• Dangers of Bayesian Model Averaging under Covariate Shift
• Deep Dirichlet uncertainty for unsupervised out-of-distribution detection of eye fundus photographs in glaucoma screening
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• A Multi-Scale Framework for Out-of-Distribution Detection in Dermoscopic Images
• Unsupervised Approaches for Out-Of-Distribution Dermoscopic Lesion Detection