Investigating the Capabilities and Limitations of Machine Learning for Identifying Bias in English Language Data with Information and Heritage Professionals

Abstract

Despite numerous efforts to mitigate their biases, ML systems continue to harm already-marginalized people. While predominant ML approaches assume bias can be removed and fair models can be created, we show that these are not always possible, nor desirable, goals. We reframe the problem of ML bias by creating models to identify biased language, drawing attention to a dataset's biases rather than trying to remove them. Then, through a workshop, we evaluated the models for a specific use case: workflows of information and heritage professionals. Our findings demonstrate the limitations of ML for identifying bias due to its contextual nature, the way in which approaches to mitigating it can simultaneously privilege and oppress different communities, and its inevitability. We demonstrate the need to expand ML approaches to bias and fairness, providing a mixed-methods approach to investigating the feasibility of removing bias or achieving fairness in a given ML use case.

Accepted to the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25)

References in corpus (12)

• Improving fairness in machine learning systems: What do industry practitioners need?
• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
• Data Feminism for AI
• Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
• Human-Centered Responsible Artificial Intelligence: Current & Future Trends
• Algorithmic Fairness and Structural Injustice: Insights from Feminist Political Philosophy
• Case Study: Predictive Fairness to Reduce Misdemeanor Recidivism Through Social Service Interventions
• Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task
• Toward Gender-Inclusive Coreference Resolution
• Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systems
• Situated Cameras, Situated Knowledges: Towards an Egocentric Epistemology for Computer Vision