Lazy Data Practices Harm Fairness Research

Abstract

Data practices shape research and practice on fairness in machine learning (fair ML). Critical data studies offer important reflections and critiques for the responsible advancement of the field by highlighting shortcomings and proposing recommendations for improvement. In this work, we present a comprehensive analysis of fair ML datasets, demonstrating how unreflective yet common practices hinder the reach and reliability of algorithmic fairness findings. We systematically study protected information encoded in tabular datasets and their usage in 280 experiments across 142 publications. Our analyses identify three main areas of concern: (1) a \textbf{lack of representation for certain protected attributes} in both data and evaluations; (2) the widespread \textbf{exclusion of minorities} during data preprocessing; and (3) \textbf{opaque data processing} threatening the generalization of fairness research. By conducting exemplary analyses on the utilization of prominent datasets, we demonstrate how unreflective data decisions disproportionately affect minority groups, fairness metrics, and resultant model comparisons. Additionally, we identify supplementary factors such as limitations in publicly available data, privacy considerations, and a general lack of awareness, which exacerbate these challenges. To address these issues, we propose a set of recommendations for data usage in fairness research centered on transparency and responsible inclusion. This study underscores the need for a critical reevaluation of data practices in fair ML and offers directions to improve both the sourcing and usage of datasets.

References in corpus (13)

• BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
• Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved
• The Forgotten Margins of AI Ethics
• Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey
• Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation
• Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis
• Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders
• WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT?
• Four Years of FAccT: A Reflexive, Mixed-Methods Analysis of Research Contributions, Shortcomings, and Future Prospects
• It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making
• An Empirical Analysis of Racial Categories in the Algorithmic Fairness Literature
• Fairness Implications of Encoding Protected Categorical Attributes

Cited by in corpus (1)

• Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey