Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective

Abstract

The Rashomon effect presents a significant challenge in model selection. It occurs when multiple models achieve similar performance on a dataset but produce different predictions, resulting in predictive multiplicity. This is especially problematic in high-stakes environments, where arbitrary model outcomes can have serious consequences. Traditional model selection methods prioritize accuracy and fail to address this issue. Factors such as class imbalance and irrelevant variables further complicate the situation, making it harder for models to provide trustworthy predictions. Data-centric AI approaches can mitigate these problems by prioritizing data optimization, particularly through preprocessing techniques. However, recent studies suggest preprocessing methods may inadvertently inflate predictive multiplicity. This paper investigates how data preprocessing techniques like balancing and filtering methods impact predictive multiplicity and model stability, considering the complexity of the data. We conduct the experiments on 21 real-world datasets, applying various balancing and filtering techniques, and assess the level of predictive multiplicity introduced by these methods by leveraging the Rashomon effect. Additionally, we examine how filtering techniques reduce redundancy and enhance model generalization. The findings provide insights into the relationship between balancing methods, data complexity, and predictive multiplicity, demonstrating how data-centric AI strategies can improve model performance.

38 pages, 7 figures

References in corpus (11)

• SMOTE: Synthetic Minority Over-sampling Technique
• Data-centric Artificial Intelligence: A Survey
• On the Existence of Simpler Machine Learning Models
• Amazing Things Come From Having Many Good Models
• Rashomon Capacity: A Metric for Predictive Multiplicity in Classification
• Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
• An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification
• The Cost of Arbitrariness for Individuals: Examining the Legal and Technical Challenges of Model Multiplicity
• Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation
• Predictive Churn with the Set of Good Models
• forester: A Tree-Based AutoML Tool in R

Cited by in corpus (1)

• Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML