Improving fairness in machine learning systems: What do industry practitioners need?

Abstract

The potential for machine learning (ML) systems to amplify social inequities and unfairness is receiving increasing popular and academic attention. A surge of recent work has focused on the development of algorithmic tools to assess and mitigate such unfairness. If these tools are to have a positive impact on industry practice, however, it is crucial that their design be informed by an understanding of real-world needs. Through 35 semi-structured interviews and an anonymous survey of 267 ML practitioners, we conduct the first systematic investigation of commercial product teams' challenges and needs for support in developing fairer ML systems. We identify areas of alignment and disconnect between the challenges faced by industry practitioners and solutions proposed in the fair ML research literature. Based on these findings, we highlight directions for future ML and HCI research that will better address industry practitioners' needs.

To appear in the 2019 ACM CHI Conference on Human Factors in Computing Systems (CHI 2019)

References in corpus (4)

• 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
• Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making
• Fairness Testing: Testing Software for Discrimination
• Fairness in Machine Learning: Lessons from Political Philosophy

Cited by in corpus (102)

• Principles alone cannot guarantee ethical AI
• Towards Personalized Federated Learning
• Fairness And Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, And Mitigation Strategies
• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices
• Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
• Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
• Demographic Bias in Biometrics: A Survey on an Emerging Challenge
• More Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• FairSight: Visual Analytics for Fairness in Decision Making
• Dislocated Accountabilities in the AI Supply Chain: Modularity and Developers' Notions of Responsibility
• Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
• This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology
• Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook
• Interactive Model Cards: A Human-Centered Approach to Model Documentation
• Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline
• Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
• Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness
• Improving Human-AI Collaboration With Descriptions of AI Behavior
• Walking the Walk of AI Ethics: Organizational Challenges and the Individualization of Risk among Ethics Entrepreneurs
• Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
• Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
• Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
• Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation
• Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation
• Discovering and Validating AI Errors With Crowdsourced Failure Reports
• Visual Analysis of Discrimination in Machine Learning
• Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
• Towards an AI-Enhanced Cyber Threat Intelligence Processing Pipeline
• Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systems
• Equality of Learning Opportunity via Individual Fairness in Personalized Recommendations
• LiFT: A Scalable Framework for Measuring Fairness in ML Applications
• Mediating Community-AI Interaction through Situated Explanation: The Case of AI-Led Moderation
• Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey
• Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscrimination
• Understanding Collaborative Practices and Tools of Professional UX Practitioners in Software Organizations
• Seamful XAI: Operationalizing Seamful Design in Explainable AI
• Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud
• Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development
• How Do UX Practitioners Communicate AI as a Design Material? Artifacts, Conceptions, and Propositions
• Inherent Limitations of AI Fairness
• A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations
• Towards Understanding Fairness and its Composition in Ensemble Machine Learning
• Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• Aspirations and Practice of Model Documentation: Moving the Needle with Nudging and Traceability
• Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making
• Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences
• Artificial Concepts of Artificial Intelligence: Institutional Compliance and Resistance in AI Startups
• Deep Learning feature selection to unhide demographic recommender systems factors
• "We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning
• Fairness in Agreement With European Values: An Interdisciplinary Perspective on AI Regulation
• Angler: Helping Machine Translation Practitioners Prioritize Model Improvements
• Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools
• Having your Privacy Cake and Eating it Too: Platform-supported Auditing of Social Media Algorithms for Public Interest
• FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning
• How Data Scientists Review the Scholarly Literature
• Stakeholder Participation for Responsible AI Development: Disconnects Between Guidance and Current Practice
• Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness
• Machine Learning Data Practices through a Data Curation Lens: An Evaluation Framework
• Fairness Hacking: The Malicious Practice of Shrouding Unfairness in Algorithms
• Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithms
• Machine Learning practices and infrastructures
• On Prediction-Modelers and Decision-Makers: Why Fairness Requires More Than a Fair Prediction Model
• Prescriptive and Descriptive Approaches to Machine-Learning Transparency
• The Interaction Layer: An Exploration for Co-Designing User-LLM Interactions in Parental Wellbeing Support Systems
• "Come to us first": Centering Community Organizations in Artificial Intelligence for Social Good Partnerships
• Access Denied: Meaningful Data Access for Quantitative Algorithm Audits
• Analysing and Organising Human Communications for AI Fairness-Related Decisions: Use Cases from the Public Sector
• Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
• Who is Responsible When AI Fails? Mapping Causes, Entities, and Consequences of AI Privacy and Ethical Incidents
• "Is It My Turn?" Assessing Teamwork and Taskwork in Collaborative Immersive Analytics
• AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients
• Modelling the long-term fairness dynamics of data-driven targeted help on job seekers
• Integrity and Junkiness Failure Handling for Embedding-based Retrieval: A Case Study in Social Network Search
• The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research
• ScholarMate: A Mixed-Initiative Tool for Qualitative Knowledge Work and Information Sensemaking
• Towards Creating Infrastructures for Values and Ethics Work in the Production of Software Technologies
• RecRec: Algorithmic Recourse for Recommender Systems
• "I Hadn't Thought About That": Creators of Human-like AI Weigh in on Ethics And Neurodivergence
• VLSlice: Interactive Vision-and-Language Slice Discovery
• Group fairness without demographics using social networks
• Auditing for Bias in Ad Delivery Using Inferred Demographic Attributes
• Translating Emotions to Annotations -- A Participant Perspective of Physiological Emotion Data Collection
• Experts' View on Challenges and Needs for Fairness in Artificial Intelligence for Education
• Mapping Stakeholder Needs to Multi-Sided Fairness in Candidate Recommendation for Algorithmic Hiring
• Multi-Agent LLMs as Ethics Advocates for AI-Based Systems
• Human-in-the-Loop Annotation for Image-Based Engagement Estimation: Assessing the Impact of Model Reliability on Annotation Accuracy
• Towards Interactively Improving ML Data Preparation Code via "Shadow Pipelines"
• What is fair? Exploring the artists' perspective on the fairness of music streaming platforms
• Investigating the Capabilities and Limitations of Machine Learning for Identifying Bias in English Language Data with Information and Heritage Professionals
• On the Impact of Data Quality on Image Classification Fairness
• DomainDemo: a dataset of domain-sharing activities among different demographic groups on Twitter
• Algorithmic Fairness Datasets: the Story so Far
• Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
• Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software
• Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks