Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions

Abstract

We develop and rigorously evaluate a deep learning based system that can accurately classify skin conditions while detecting rare conditions for which there is not enough data available for training a confident classifier. We frame this task as an out-of-distribution (OOD) detection problem. Our novel approach, hierarchical outlier detection (HOD) assigns multiple abstention classes for each training outlier class and jointly performs a coarse classification of inliers vs. outliers, along with fine-grained classification of the individual classes. We demonstrate the effectiveness of the HOD loss in conjunction with modern representation learning approaches (BiT, SimCLR, MICLe) and explore different ensembling strategies for further improving the results. We perform an extensive subgroup analysis over conditions of varying risk levels and different skin types to investigate how the OOD detection performance changes over each subgroup and demonstrate the gains of our framework in comparison to baselines. Finally, we introduce a cost metric to approximate downstream clinical impact. We use this cost metric to compare the proposed method against a baseline system, thereby making a stronger case for the overall system effectiveness in a real-world deployment scenario.

Under Review, 19 Pages

References in corpus (7)

• Explaining and Harnessing Adversarial Examples
• Contrastive Training for Improved Out-of-Distribution Detection
• A Benchmark of Medical Out of Distribution Detection
• Supervised Transfer Learning at Scale for Medical Imaging
• Calibrating Healthcare AI: Towards Reliable and Interpretable Deep Predictive Models
• Self-Supervised Out-of-Distribution Detection in Brain CT Scans
• CheXseen: Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays

Cited by in corpus (8)

• Recommendations on test datasets for evaluating AI solutions in pathology
• Learning Optimal Conformal Classifiers
• Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection
• Dangers of Bayesian Model Averaging under Covariate Shift
• Deep Dirichlet uncertainty for unsupervised out-of-distribution detection of eye fundus photographs in glaucoma screening
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• A Multi-Scale Framework for Out-of-Distribution Detection in Dermoscopic Images
• Unsupervised Approaches for Out-Of-Distribution Dermoscopic Lesion Detection