Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs
arXiv:2411.04994 · doi:10.1145/3715275.3732049
Abstract
Most AI tools adopted by governments are not developed internally, but instead are acquired from third-party vendors in a process called public procurement. In this paper, we conduct the first empirical study of how United States cities' procurement practices shape critical decisions surrounding public sector AI. We conduct semi-structured interviews with 19 city employees who oversee AI procurement across 7 U.S. cities. We found that cities' legacy procurement practices, which are shaped by decades-old laws and norms, establish infrastructure that determines which AI is purchased, and which actors hold decision-making power over procured AI. We characterize the emerging actions cities have taken to adapt their purchasing practices to address algorithmic harms. From employees' reflections on real-world AI procurements, we identify three key challenges that motivate but are not fully addressed by existing AI procurement reform initiatives. Based on these findings, we discuss implications and opportunities for the FAccT community to support cities in foreseeing and preventing AI harms throughout the public procurement processes.
10 pages, 2 column format. In proceedings of ACM FAccT 2025
References in corpus (14)
- Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making
- The Fallacy of AI Functionality
- Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
- Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
- Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics
- Algorithms and Decision-Making in the Public Sector
- Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Services
- Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders
- Black-Box Access is Insufficient for Rigorous AI Audits
- AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries
- Modeling Assumptions Clash with the Real World: Transparency, Equity, and Community Challenges for Student Assignment Algorithms
- The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposals
- The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment
- Public Technologies Transforming Work of the Public and the Public Sector