Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders

Abstract

Child welfare agencies across the United States are turning to data-driven predictive technologies (commonly called predictive analytics) which use government administrative data to assist workers' decision-making. While some prior work has explored impacted stakeholders' concerns with current uses of data-driven predictive risk models (PRMs), less work has asked stakeholders whether such tools ought to be used in the first place. In this work, we conducted a set of seven design workshops with 35 stakeholders who have been impacted by the child welfare system or who work in it to understand their beliefs and concerns around PRMs, and to engage them in imagining new uses of data and technologies in the child welfare system. We found that participants worried current PRMs perpetuate or exacerbate existing problems in child welfare. Participants suggested new ways to use data and data-driven tools to better support impacted communities and suggested paths to mitigate possible harms of these tools. Participants also suggested low-tech or no-tech alternatives to PRMs to address problems in child welfare. Our study sheds light on how researchers and designers can work in solidarity with impacted communities, possibly to circumvent or oppose child welfare agencies.

10 pages in main body; 4 pages in appendix; Published in the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT'22)

References in corpus (4)

• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
• Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems
• Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systems
• How to Train a (Bad) Algorithmic Caseworker: A Quantitative Deconstruction of Risk Assessments in Child-Welfare

Cited by in corpus (15)

• Harms from Increasingly Agentic Algorithmic Systems
• Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
• The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposals
• Stakeholder-Centered AI Design: Co-Designing Worker Tools with Gig Workers through Data Probes
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policing
• The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment
• LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions
• The AI-DEC: A Card-based Design Method for User-centered AI Explanations
• Lazy Data Practices Harm Fairness Research
• Secondary Stakeholders in AI: Fighting for, Brokering, and Navigating Agency
• Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs
• Public Technologies Transforming Work of the Public and the Public Sector
• The Datafication of Care in Public Homelessness Services
• Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making