Classifier Technology and the Illusion of Progress

Abstract

A great many tools have been developed for supervised classification, ranging from early methods such as linear discriminant analysis through to modern developments such as neural networks and support vector machines. A large number of comparative studies have been conducted in attempts to establish the relative superiority of these methods. This paper argues that these comparisons often fail to take into account important aspects of real problems, so that the apparent superiority of more sophisticated methods may be something of an illusion. In particular, simple methods typically yield performance almost as good as more sophisticated methods, to the extent that the difference in performance may be swamped by other sources of uncertainty that generally are not considered in the classical supervised classification paradigm.

This paper commented in: [math.ST/0606447], [math.ST/0606452], [math.ST/0606455], [math.ST/0606457]. Rejoinder in [math.ST/0606461]. Published at http://dx.doi.org/10.1214/088342306000000060 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Cited by in corpus (43)

• OpenML: networked science in machine learning
• Data Mining and Machine Learning in Astronomy
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Learning under Concept Drift: an Overview
• Liquid chromatography mass spectrometry-based proteomics: Biological and technological aspects
• Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification
• Gene ranking and biomarker discovery under correlation
• Feature selection in omics prediction problems using cat scores and false nondiscovery rate control
• Verbal Autopsy Methods with Multiple Causes of Death
• A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint
• Data Science vs. Statistics: Two Cultures?
• Maximin effects in inhomogeneous large-scale data
• Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
• The mRMR variable selection method: a comparative study for functional data
• Data Mining and Electronic Health Records: Selecting Optimal Clinical Treatments in Practice
• Data Mining Session-Based Patient Reported Outcomes (PROs) in a Mental Health Setting: Toward Data-Driven Clinical Decision Support and Personalized Treatment
• On the dimension effect of regularized linear discriminant analysis
• Recognition of basic hand movements using Electromyography
• Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints
• Subgroup detection in linear growth curve models with generalized linear mixed model (GLMM) trees
• Bayesian Synthesis: Combining subjective analyses, with an application to ozone data
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Forks Over Knives: Predictive Inconsistency in Criminal Justice Algorithmic Risk Assessment Tools
• Supervised functional classification: A theoretical remark and some comparisons
• Storms prediction : Logistic regression vs random forest for unbalanced data
• An evaluation of machine learning techniques to predict the outcome of children treated for Hodgkin-Lymphoma on the AHOD0031 trial: A report from the Children's Oncology Group
• Integrative genetic risk prediction using nonparametric empirical Bayes classification
• Algorithmic insights on continual learning from fruit flies
• Progressing Towards Responsible AI
• Diagnosing the performance of human mobility models at small spatial scales using volunteered geographic information
• Credit Scoring for Micro-Loans
• Wikistat 2.0: Educational Resources for Artificial Intelligence
• TULIP: A Toolbox for Linear Discriminant Analysis with Penalties
• Critères de qualité d'un classifieur généraliste
• On the variability of regression shrinkage methods for clinical prediction models: simulation study on predictive performance
• Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
• A partition-based similarity for classification distributions
• Treatment Effect Bias from Sample Snooping: Blinding Outcomes is Neither Necessary nor Sufficient
• Minimax Optimal Estimation of Stability Under Distribution Shift
• Learning Optimal Distributionally Robust Individualized Treatment Rules
• Personalised dynamic super learning: an application in predicting hemodiafiltration convection volumes
• A Hybrid Statistical Learning Framework for Capability-Based Decision Support in Manufacturing
• The Interaction of Entropy-Based Discretization and Sample Size: An Empirical Study