Sample Size Planning for Classification Models

Abstract

In biospectroscopy, suitably annotated and statistically independent samples (e. g. patients, batches, etc.) for classifier training and testing are scarce and costly. Learning curves show the model performance as function of the training sample size and can help to determine the sample size needed to train good classifiers. However, building a good model is actually not enough: the performance must also be proven. We discuss learning curves for typical small sample size situations with 5 - 25 independent samples per class. Although the classification models achieve acceptable performance, the learning curve can be completely masked by the random testing uncertainty due to the equally limited test sample size. In consequence, we determine test sample sizes necessary to achieve reasonable precision in the validation and find that 75 - 100 samples will usually be needed to test a good but not perfect classifier. Such a data set will then allow refined sample size planning on the basis of the achieved performance. We also demonstrate how to calculate necessary sample sizes in order to show the superiority of one classifier over another: this often requires hundreds of statistically independent test samples or is even theoretically impossible. We demonstrate our findings with a data set of ca. 2550 Raman spectra of single cells (five classes: erythrocytes, leukocytes and three tumour cell lines BT-20, MCF-7 and OCI-AML3) as well as by an extensive simulation that allows precise determination of the actual performance of the models in question.

The paper is published in Analytica Chimica Acta (special issue "CAC2012"). This is a reformatted version of the accepted manuscript with few typos corrected and links to the official publicaion, including the supplementary material (pages 11 - 16 and supplementary-* files in the source). The slides of the presentation at Clircon (2015-04-22, Exeter, UK) are available as ancillary pdf file

Cited by in corpus (20)

• Machine Learning and Deep Learning Algorithms for Bearing Fault Diagnostics -- A Comprehensive Review
• Tractography and machine learning: Current state and open challenges
• Boosting Operational DNN Testing Efficiency through Conditioning
• $b\to c\tau {\nu }_{\tau }$ Decays: A Catalogue to Compare, Constrain, and Correlate New Physics Effects
• Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to the Grading of Astrocytoma Tissues
• Learning Curves for Decision Making in Supervised Machine Learning: A Survey
• Exhaustive Model Selection in $b\to s\ell \ell$ Decays: Pitting Cross-Validation against AIC${}_c$
• A novel fusion of Sentinel-1 and Sentinel-2 with climate data for crop phenology estimation using Machine Learning
• Supervised machine learning for microbiomics: bridging the gap between current and best practices
• Large-scale comparative visualisation of sets of multidimensional data
• Identifying New X-ray Binary Candidates in M31 using Random Forest Classification
• Sample size determination via learning-type curves
• Reconfigurable Intelligent Surface Assisted Mobile Edge Computing with Heterogeneous Learning Tasks
• Machine Learning Approaches to Automated Flow Cytometry Diagnosis of Chronic Lymphocytic Leukemia
• Rethinking Generalization Performance of Surgical Phase Recognition with Expert-Generated Annotations
• Interpretable feature subset selection: A Shapley value based approach
• Detection of fake news on CoViD-19 on Web Search Engines
• A Generative Neural Network Framework for Automated Software Testing
• A Simple and Scalable Kernel Density Approach for Reliable Uncertainty Quantification in Atomistic Machine Learning
• Reconfigurable Intelligent Surface Assisted Edge Machine Learning