Should we really use post-hoc tests based on mean-ranks?

Abstract

The statistical comparison of multiple algorithms over multiple data sets is fundamental in machine learning. This is typically carried out by the Friedman test. When the Friedman test rejects the null hypothesis, multiple comparisons are carried out to establish which are the significant differences among algorithms. The multiple comparisons are usually performed using the mean-ranks test. The aim of this technical note is to discuss the inconsistencies of the mean-ranks post-hoc test with the goal of discouraging its use in machine learning as well as in medicine, psychology, etc.. We show that the outcome of the mean-ranks test depends on the pool of algorithms originally included in the experiment. In other words, the outcome of the comparison between algorithms A and B depends also on the performance of the other algorithms included in the original experiment. This can lead to paradoxical situations. For instance the difference between A and B could be declared significant if the pool comprises algorithms C, D, E and not significant if the pool comprises algorithms F, G, H. To overcome these issues, we suggest instead to perform the multiple comparison using a test whose outcome only depends on the two algorithms being compared, such as the sign-test or the Wilcoxon signed-rank test.

Cited by in corpus (36)

• Deep learning for time series classification: a review
• How I failed machine learning in medical imaging -- shortcomings and recommendations
• Benchmarking Multivariate Time Series Classification Algorithms
• MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification
• HIVE-COTE 2.0: a new meta ensemble for time series classification
• The Heterogeneous Ensembles of Standard Classification Algorithms (HESCA): the Whole is Greater than the Sum of its Parts
• Deep Neural Network Ensembles for Time Series Classification
• Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis
• Scalable Dictionary Classifiers for Time Series Classification
• Time-Series Classification in Smart Manufacturing Systems: An Experimental Evaluation of State-of-the-Art Machine Learning Algorithms
• From BOP to BOSS and Beyond: Time Series Classification with Dictionary Based Classifiers
• FIRE-DES++: Enhanced Online Pruning of Base Classifiers for Dynamic Ensemble Selection
• Ensembles of Localised Models for Time Series Forecasting
• Multi-Label Classification Neural Networks with Hard Logical Constraints
• Is rotation forest the best classifier for problems with continuous features?
• Multiple Similarity Drug-Target Interaction Prediction with Random Walks and Matrix Factorization
• On the Use of Default Parameter Settings in the Empirical Evaluation of Classification Algorithms
• Average Localised Proximity: A new data descriptor with good default one-class classification performance
• Making Classifier Chains Resilient to Class Imbalance
• K-Nearest Oracles Borderline Dynamic Classifier Ensemble Selection
• NILMFormer: Non-Intrusive Load Monitoring that Accounts for Non-Stationarity
• Coherent Hierarchical Multi-Label Classification Networks
• Improving Problem Identification via Automated Log Clustering using Dimensionality Reduction
• A Generalised Signature Method for Multivariate Time Series Feature Extraction
• Investigating the Effectiveness of Representations Based on Word-Embeddings in Active Learning for Labelling Text Datasets
• QCBA: Improving Rule Classifiers Learned from Quantitative Data by Recovering Information Lost by Discretisation
• Synthetic Oversampling of Multi-Label Data based on Local Label Distribution
• Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
• An interpretable semi-supervised classifier using two different strategies for amended self-labeling
• Multi-Label Sampling based on Local Label Imbalance
• Online local pool generation for dynamic classifier selection: an extended version
• Learnable Dynamic Temporal Pooling for Time Series Classification
• A tale of two toolkits, report the second: bake off redux. Chapter 1. dictionary based classifiers
• Video Scene Location Recognition with Neural Networks
• Short-term Cognitive Networks, Flexible Reasoning and Nonsynaptic Learning
• Model Compression for Dynamic Forecast Combination