Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy

Abstract

Diabetic retinopathy (DR) and diabetic macular edema are common complications of diabetes which can lead to vision loss. The grading of DR is a fairly complex process that requires the detection of fine features such as microaneurysms, intraretinal hemorrhages, and intraretinal microvascular abnormalities. Because of this, there can be a fair amount of grader variability. There are different methods of obtaining the reference standard and resolving disagreements between graders, and while it is usually accepted that adjudication until full consensus will yield the best reference standard, the difference between various methods of resolving disagreements has not been examined extensively. In this study, we examine the variability in different methods of grading, definitions of reference standards, and their effects on building deep learning models for the detection of diabetic eye disease. We find that a small set of adjudicated DR grades allows substantial improvements in algorithm performance. The resulting algorithm's performance was on par with that of individual U.S. board-certified ophthalmologists and retinal specialists.

Cited by in corpus (39)

• Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• DeepSeeNet: A deep learning model for automated classification of patient-based age-related macular degeneration severity from color fundus photographs
• Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs
• Detecting Anemia from Retinal Fundus Images
• Deep Learning and Glaucoma Specialists: The Relative Importance of Optic Disc Features to Predict Glaucoma Referral in Fundus Photos
• Predicting optical coherence tomography-derived diabetic macular edema grades from fundus photographs using deep learning
• DR$|$GRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images
• Active label cleaning for improved dataset quality under resource constraints
• A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision
• Classification of crystallization outcomes using deep convolutional neural networks
• Dual Branch Deep Learning Network for Detection and Stage Grading of Diabetic Retinopathy
• Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
• Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)
• An Interpretable Multiple-Instance Approach for the Detection of referable Diabetic Retinopathy from Fundus Images
• Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders
• Learning Robust Representation for Joint Grading of Ophthalmic Diseases via Adaptive Curriculum and Feature Disentanglement
• MAPLES-DR: MESSIDOR Anatomical and Pathological Labels for Explainable Screening of Diabetic Retinopathy
• Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond
• VR-FuseNet: A Fusion of Heterogeneous Fundus Data and Explainable Deep Network for Diabetic Retinopathy Classification
• Explaining in Style: Training a GAN to explain a classifier in StyleSpace
• Predicting Progression of Age-related Macular Degeneration from Fundus Images using Deep Learning
• Balanced-MixUp for Highly Imbalanced Medical Image Classification
• AGE Challenge: Angle Closure Glaucoma Evaluation in Anterior Segment Optical Coherence Tomography
• Adversarial Exposure Attack on Diabetic Retinopathy Imagery Grading
• Explainable Autonomous Robots: A Survey and Perspective
• Unsupervised deep learning for grading of age-related macular degeneration using retinal fundus images
• Applications of Deep Learning in Fundus Images: A Review
• Towards Assessing Data Bias in Clinical Trials
• Are Ensemble Classifiers Powerful Enough for the Detection and Diagnosis of Intermediate-Severity Faults?
• Improving Medical Annotation Quality to Decrease Labeling Burden Using Stratified Noisy Cross-Validation
• Information-theoretic Classification Accuracy: A Criterion that Guides Data-driven Combination of Ambiguous Outcome Labels in Multi-class Classification
• Using Ensemble Classifiers to Detect Incipient Anomalies
• Artificial Intelligence for Pediatric Ophthalmology
• A multi-center prospective evaluation of THEIA to detect diabetic retinopathy (DR) and diabetic macular edema (DME) in the New Zealand screening program
• CANet: Cross-disease Attention Network for Joint Diabetic Retinopathy and Diabetic Macular Edema Grading
• A Framework of Explanation Generation toward Reliable Autonomous Robots