REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs

Abstract

Glaucoma is one of the leading causes of irreversible but preventable blindness in working age populations. Color fundus photography (CFP) is the most cost-effective imaging modality to screen for retinal disorders. However, its application to glaucoma has been limited to the computation of a few related biomarkers such as the vertical cup-to-disc ratio. Deep learning approaches, although widely applied for medical image analysis, have not been extensively used for glaucoma assessment due to the limited size of the available data sets. Furthermore, the lack of a standardize benchmark strategy makes difficult to compare existing methods in a uniform way. In order to overcome these issues we set up the Retinal Fundus Glaucoma Challenge, REFUGE (\url{https://refuge.grand-challenge.org}), held in conjunction with MICCAI 2018. The challenge consisted of two primary tasks, namely optic disc/cup segmentation and glaucoma classification. As part of REFUGE, we have publicly released a data set of 1200 fundus images with ground truth segmentations and clinical glaucoma labels, currently the largest existing one. We have also built an evaluation framework to ease and ensure fairness in the comparison of different models, encouraging the development of novel techniques in the field. 12 teams qualified and participated in the online challenge. This paper summarizes their methods and analyzes their corresponding results. In particular, we observed that two of the top-ranked teams outperformed two human experts in the glaucoma classification task. Furthermore, the segmentation results were in general consistent with the ground truth annotations, with complementary outcomes that can be further exploited by ensembling the results.

Accepted for publication in Medical Image Analysis

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SMOTE: Synthetic Minority Over-sampling Technique
• A Survey on Deep Learning in Medical Image Analysis
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning
• Joint Optic Disc and Cup Segmentation Based on Multi-label Deep Network and Polar Transformation
• Optic Disc and Cup Segmentation Methods for Glaucoma Detection with Modification of U-Net Convolutional Neural Network
• Why rankings of biomedical image analysis competitions should be interpreted with care
• Patch-based Output Space Adversarial Learning for Joint Optic Disc and Cup Segmentation
• Deep Learning and Glaucoma Specialists: The Relative Importance of Optic Disc Features to Predict Glaucoma Referral in Fundus Photos
• Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks
• Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation
• Stack-U-Net: Refinement Network for Image Segmentation on the Example of Optic Disc and Cup
• Classification of Findings with Localized Lesions in Fundoscopic Images using a Regionally Guided CNN

Cited by in corpus (49)

• Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
• DoFE: Domain-oriented Feature Embedding for Generalizable Fundus Image Segmentation on Unseen Datasets
• Transformers in Healthcare: A Survey
• Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
• Recent Progress in Transformer-based Medical Image Analysis
• Deep learning on fundus images detects glaucoma beyond the optic disc
• A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision
• AADG: Automatic Augmentation for Domain Generalization on Retinal Image Segmentation
• ADAM Challenge: Detecting Age-related Macular Degeneration from Fundus Images
• Deep Triplet Hashing Network for Case-based Medical Image Retrieval
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• Affinity Space Adaptation for Semantic Segmentation Across Domains
• FundusQ-Net: a Regression Quality Assessment Deep Learning Algorithm for Fundus Images Quality Grading
• FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space
• Transformation Consistent Self-ensembling Model for Semi-supervised Medical Image Segmentation
• Task Decomposition and Synchronization for Semantic Biomedical Image Segmentation
• Medical Image Segmentation Using Squeeze-and-Expansion Transformers
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images
• MAPLES-DR: MESSIDOR Anatomical and Pathological Labels for Explainable Screening of Diabetic Retinopathy
• Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks
• BI-GCN: Boundary-Aware Input-Dependent Graph Convolution Network for Biomedical Image Segmentation
• AGE Challenge: Angle Closure Glaucoma Evaluation in Anterior Segment Optical Coherence Tomography
• A Systematic Collection of Medical Image Datasets for Deep Learning
• Modeling and Enhancing Low-quality Retinal Fundus Images
• VascX Models: Model Ensembles for Retinal Vascular Analysis from Color Fundus Images
• U-Net with Hierarchical Bottleneck Attention for Landmark Detection in Fundus Images of the Degenerated Retina
• Rapid Image Labeling via Neuro-Symbolic Learning
• Regression and Learning with Pixel-wise Attention for Retinal Fundus Glaucoma Segmentation and Detection
• M3Lung-Sys: A Deep Learning System for Multi-Class Lung Pneumonia Screening from CT Imaging
• Applications of Deep Learning in Fundus Images: A Review
• Deep Dirichlet uncertainty for unsupervised out-of-distribution detection of eye fundus photographs in glaucoma screening
• Self-supervised learning via inter-modal reconstruction and feature projection networks for label-efficient 3D-to-2D segmentation
• Source-Free Domain Adaptive Fundus Image Segmentation with Denoised Pseudo-Labeling
• Semi-weakly Supervised Contrastive Representation Learning for Retinal Fundus Images
• Self-Adaptive Transfer Learning for Multicenter Glaucoma Classification in Fundus Retina Images
• Difficulty-aware Glaucoma Classification with Multi-Rater Consensus Modeling
• Semi-supervised learning for medical image classification using imbalanced training data
• Meta-learners for few-shot weakly-supervised optic disc and cup segmentation on fundus images
• MI^2GAN: Generative Adversarial Network for Medical Image Domain Adaptation using Mutual Information Constraint
• Medical Datasets Collections for Artificial Intelligence-based Medical Image Analysis
• Few-Shot Domain Adaptation with Polymorphic Transformers
• Uncertainty-Supervised Interpretable and Robust Evidential Segmentation
• Quasi-multimodal-based pathophysiological feature learning for retinal disease diagnosis
• Contour Flow Constraint: Preserving Global Shape Similarity for Deep Learning based Image Segmentation
• Fundus Image Analysis for Age Related Macular Degeneration: ADAM-2020 Challenge Report
• Soft Dice Confidence: A Near-Optimal Confidence Estimator for Selective Prediction in Semantic Segmentation
• FunduSegmenter: Leveraging the RETFound Foundation Model for Joint Optic Disc and Optic Cup Segmentation in Retinal Fundus Images
• Multiscale Softmax Cross Entropy for Fovea Localization on Color Fundus Photography