Towards a multi-stakeholder value-based assessment framework for algorithmic systems
arXiv:2205.04525 · doi:10.1145/3531146.3533118
Abstract
In an effort to regulate Machine Learning-driven (ML) systems, current auditing processes mostly focus on detecting harmful algorithmic biases. While these strategies have proven to be impactful, some values outlined in documents dealing with ethics in ML-driven systems are still underrepresented in auditing processes. Such unaddressed values mainly deal with contextual factors that cannot be easily quantified. In this paper, we develop a value-based assessment framework that is not limited to bias auditing and that covers prominent ethical principles for algorithmic systems. Our framework presents a circular arrangement of values with two bipolar dimensions that make common motivations and potential tensions explicit. In order to operationalize these high-level principles, values are then broken down into specific criteria and their manifestations. However, some of these value-specific criteria are mutually exclusive and require negotiation. As opposed to some other auditing frameworks that merely rely on ML researchers' and practitioners' input, we argue that it is necessary to include stakeholders that present diverse standpoints to systematically negotiate and consolidate value and criteria tensions. To that end, we map stakeholders with different insight needs, and assign tailored means for communicating value manifestations to them. We, therefore, contribute to current ML auditing practices with an assessment framework that visualizes closeness and tensions between values and we give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.
Accepted at the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT'22)
References in corpus (10)
- Questioning the AI: Informing Design Practices for Explainable AI User Experiences
- 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
- Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
- Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches
- Designing for Human Rights in AI
- Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
- Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
- Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
- An Agent-based Model to Evaluate Interventions on Online Dating Platforms to Decrease Racial Homogamy
- Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?