Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making

Abstract

Calls for heightened consideration of fairness and accountability in algorithmically-informed public decisions---like taxation, justice, and child protection---are now commonplace. How might designers support such human values? We interviewed 27 public sector machine learning practitioners across 5 OECD countries regarding challenges understanding and imbuing public values into their work. The results suggest a disconnect between organisational and institutional realities, constraints and needs, and those addressed by current research into usable, transparent and 'discrimination-aware' machine learning---absences likely to undermine practical initiatives unless addressed. We see design opportunities in this disconnect, such as in supporting the tracking of concept drift in secondary data sources, and in building usable transparency tools to identify risks and incorporate domain knowledge, aimed both at managers and at the 'street-level bureaucrats' on the frontlines of public service. We conclude by outlining ethical challenges and future directions for collaboration in these high-stakes applications.

arXiv comment14 pages, 0 figures, ACM Conference on Human Factors in Computing Systems (CHI'18), April 21--26, Montreal, Canada

References in corpus (3)

• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Supersparse Linear Integer Models for Optimized Medical Scoring Systems
• Runaway Feedback Loops in Predictive Policing

Cited by in corpus (55)

• Improving fairness in machine learning systems: What do industry practitioners need?
• Fairness in Machine Learning: A Survey
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI Relationships
• Algorithms and Decision-Making in the Public Sector
• Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook
• Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
• Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems
• The State of Human-centered NLP Technology for Fact-checking
• Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
• Fair Regression: Quantitative Definitions and Reduction-based Algorithms
• Explainable AI, but explainable to whom?
• Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation
• Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systems
• The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposals
• Mediating Community-AI Interaction through Situated Explanation: The Case of AI-Led Moderation
• Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscrimination
• Contestable Camera Cars: A Speculative Design Exploration of Public AI That Is Open and Responsive to Dispute
• A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations
• Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysis
• Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education?
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• Factors Influencing Perceived Fairness in Algorithmic Decision-Making: Algorithm Outcomes, Development Procedures, and Individual Differences
• Implementing Responsible AI: Tensions and Trade-Offs Between Ethics Aspects
• Fairness in Agreement With European Values: An Interdisciplinary Perspective on AI Regulation
• Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policing
• Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learning
• "What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairness
• FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning
• Trustworthy AI in practice: an analysis of practitioners' needs and challenges
• ESR: Ethics and Society Review of Artificial Intelligence Research
• Capuchin: Causal Database Repair for Algorithmic Fairness
• Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness
• From Prisons to Programming: Fostering Self-Efficacy via Virtual Web Design Curricula in Prisons and Jails
• Machine Learning practices and infrastructures
• Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs
• Linguistically Differentiating Acts and Recalls of Racial Microaggressions on Social Media
• More Than Accuracy: Towards Trustworthy Machine Learning Interfaces for Object Recognition
• AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients
• The Datafication of Care in Public Homelessness Services
• Improving Fairness in Adaptive Social Exergames via Shapley Bandits
• Design of a Privacy-Preserving Data Platform for Collaboration Against Human Trafficking
• Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives
• Evidence-based explanation to promote fairness in AI systems
• Experts' View on Challenges and Needs for Fairness in Artificial Intelligence for Education
• Modeling Disclosive Transparency in NLP Application Descriptions
• Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context
• Justified Evidence Collection for Argument-based AI Fairness Assurance
• When the Oracle Misleads: Modeling the Consequences of Using Observable Rather than Potential Outcomes in Risk Assessment Instruments
• Nose to Glass: Looking In to Get Beyond
• Towards Integrating Fairness Transparently in Industrial Applications
• A Sociotechnical View of Algorithmic Fairness
• Conceptualising Contestability: Perspectives on Contesting Algorithmic Decisions