Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets
arXiv:2401.14497 · doi:10.1038/s41597-025-04382-5
Abstract
The remarkable progress of deep learning in dermatological tasks has brought us closer to achieving diagnostic accuracies comparable to those of human experts. However, while large datasets play a crucial role in the development of reliable deep neural network models, the quality of data therein and their correct usage are of paramount importance. Several factors can impact data quality, such as the presence of duplicates, data leakage across train-test partitions, mislabeled images, and the absence of a well-defined test partition. In this paper, we conduct meticulous analyses of three popular dermatological image datasets: DermaMNIST, its source HAM10000, and Fitzpatrick17k, uncovering these data quality issues, measure the effects of these problems on the benchmark results, and propose corrections to the datasets. Besides ensuring the reproducibility of our analysis, by making our analysis pipeline and the accompanying code publicly available, we aim to encourage similar explorations and to facilitate the identification and addressing of potential data quality issues in other large datasets.
41 pages, 17 figures, 4 tables
References in corpus (18)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis
- BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
- A Survey on Deep Learning for Skin Lesion Segmentation
- Quantum Vision Transformers
- Improving dermatology classifiers across populations using images generated by large diffusion models
- Achieve Fairness without Demographics for Dermatological Disease Diagnosis
- How You Split Matters: Data Leakage and Subject Characteristics Studies in Longitudinal Brain MRI Analysis
- GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
- Adversarial Auto-Augment with Label Preservation: A Representation Learning Principle Guided Approach
- Towards Reliable Dermatology Evaluation Benchmarks
- SkinGEN: an Explainable Dermatology Diagnosis-to-Generation Framework with Interactive Vision-Language Models
- Accelerating Batch Active Learning Using Continual Learning Techniques
- MedMNIST-C: Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions
- Synthetic Generation of Dermatoscopic Images with GAN and Closed-Form Factorization
- A Multimodal Vision Foundation Model for Clinical Dermatology