Data, Depth, and Design: Learning Reliable Models for Skin Lesion Analysis
arXiv:1711.00441 · doi:10.1016/j.neucom.2019.12.003
Abstract
Deep learning fostered a leap ahead in automated skin lesion analysis in the last two years. Those models are expensive to train and difficult to parameterize. Objective: We investigate methodological issues for designing and evaluating deep learning models for skin lesion analysis. We explore 10 choices faced by researchers: use of transfer learning, model architecture, train dataset, image resolution, type of data augmentation, input normalization, use of segmentation, duration of training, additional use of SVMs, and test data augmentation. Methods: We perform two full factorial experiments, for five different test datasets, resulting in 2560 exhaustive trials in our main experiment, and 1280 trials in our assessment of transfer learning. We analyze both with multi-way ANOVA. We use the exhaustive trials to simulate sequential decisions and ensembles, with and without the use of privileged information from the test set. Results -- main experiment: Amount of train data has disproportionate influence, explaining almost half the variation in performance. Of the other factors, test data augmentation and input resolution are the most influential. Deeper models, when combined, with extra data, also help. -- transfer experiment: Transfer learning is critical, its absence brings huge performance penalties. -- simulations: Ensembles of models are the best option to provide reliable results with limited resources, without using privileged information and sacrificing methodological rigor. Conclusions and Significance: Advancing research on automated skin lesion analysis requires curating larger public datasets. Indirect use of privileged information from the test set to design the models is a subtle, but frequent methodological mistake that leads to overoptimistic results. Ensembles of models are a cost-effective alternative to the expensive full-factorial and to the unstable sequential designs.
arXiv comment12 pages, 6 figures, 3 tables. Article accepted at Neurocomputing
References in corpus (12)
- A Survey on Deep Learning in Medical Image Analysis
- Skin lesion detection based on an ensemble of deep convolutional neural network
- Data Augmentation for Skin Lesion Analysis
- Automatic Skin Lesion Analysis using Large-scale Dermoscopy Images and Deep Residual Networks
- Image Classification of Melanoma, Nevus and Seborrheic Keratosis by Deep Neural Network Ensemble
- RECOD Titans at ISIC Challenge 2017
- A Novel Multi-task Deep Learning Model for Skin Lesion Segmentation and Classification
- Incorporating the Knowledge of Dermatologists to Convolutional Neural Networks for the Diagnosis of Skin Lesions
- Skin Lesion Classification Using Deep Multi-scale Convolutional Neural Networks
- Towards Automated Melanoma Screening: Proper Computer Vision & Reliable Results
- Convolutional Neural Network Committees for Melanoma Classification with Classical And Expert Knowledge Based Image Transforms Data Augmentation
- Skin Lesion Classification using Class Activation Map
Cited by in corpus (4)
- Skin Lesion Synthesis with Generative Adversarial Networks
- Deep-Learning Ensembles for Skin-Lesion Segmentation, Analysis, Classification: RECOD Titans at ISIC Challenge 2018
- (De)Constructing Bias on Skin Lesion Datasets
- Data Augmentation for Skin Lesion using Self-Attention based Progressive Generative Adversarial Network