Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods
arXiv:2607.26625
The paper examines how different train‑test splitting strategies impact model evaluation in automated machine learning, comparing five methods on UCI benchmarks and introducing an Optimised‑Distribution approach that maximizes statistical similarity between splits.
Abstract
Accurate model evaluation in machine learning depends critically on how datasets are split into training and testing subsets. Standard random splitting assumes that both partitions share the same underlying distribution, an assumption often violated in datasets with class imbalance, natural clustering, or spatial autocorrelation. This paper investigates the role of statistical similarity in train-test splitting and its consequences for AutoML model evaluation. Five established strategies are compared across fifteen UCI benchmark datasets: random splitting, stratified sampling, Kennard-Stone, Duplex, and SPXY. Similarity is assessed using chi-square, Kolmogorov-Smirnov, and Maximum Mean Discrepancy (MMD) tests. Geometry-based methods consistently produce near-zero MMD scores, introducing instability into downstream performance estimates. The proposed Optimised-Distribution method treats similarity as an explicit optimisation objective and achieves the highest mean MMD similarity, 89.0%, across all strategies evaluated.
Pre-review version of a paper accepted at UKCI 2026