Decision Provenance: Harnessing data flow for accountable systems

Abstract

Demand is growing for more accountability regarding the technological systems that increasingly occupy our world. However, the complexity of many of these systems - often systems-of-systems - poses accountability challenges. A key reason for this is because the details and nature of the information flows that interconnect and drive systems, which often occur across technical and organisational boundaries, tend to be invisible or opaque. This paper argues that data provenance methods show much promise as a technical means for increasing the transparency of these interconnected systems. Specifically, given the concerns regarding ever-increasing levels of automated and algorithmic decision-making, and so-called 'algorithmic systems' in general, we propose decision provenance as a concept showing much promise. Decision provenance entails using provenance methods to provide information exposing decision pipelines: chains of inputs to, the nature of, and the flow-on effects from the decisions and actions taken (at design and run-time) throughout systems. This paper introduces the concept of decision provenance, and takes an interdisciplinary (tech-legal) exploration into its potential for assisting accountability in algorithmic systems. We argue that decision provenance can help facilitate oversight, audit, compliance, risk mitigation, and user empowerment, and we also indicate the implementation considerations and areas for research necessary for realising its vision. More generally, we make the case that considerations of data flow, and systems more broadly, are important to discussions of accountability, and complement the considerable attention already given to algorithmic specifics.

Published in IEEE Access, vol. 9, pp. 6562-6574, 2019

References in corpus (4)

• European Union regulations on algorithmic decision-making and a "right to explanation"
• Practical Whole-System Provenance Capture
• CamFlow: Managed Data-sharing for Cloud Services
• FRAPpuccino: Fault-detection through Runtime Analysis of Provenance

Cited by in corpus (17)

• Algorithms that Remember: Model Inversion Attacks and Data Protection Law
• Understanding accountability in algorithmic supply chains
• "Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?
• Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"
• Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
• A Room With an Overview: Towards Meaningful Transparency for the Consumer Internet of Things
• Trusted Multi-Party Computation and Verifiable Simulations: A Scalable Blockchain Approach
• The "Who", "What", and "How" of Responsible AI Governance: A Systematic Review and Meta-Analysis of (Actor, Stage)-Specific Tools
• Towards Traceability in Data Ecosystems using a Bill of Materials Model
• Trustworthy AI in the Age of Pervasive Computing and Big Data
• Machine Learning Systems: A Survey from a Data-Oriented Perspective
• Trustworthy, responsible, ethical AI in manufacturing and supply chains: synthesis and emerging research questions
• Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context
• Provenance-Based Interpretation of Multi-Agent Information Analysis
• Expressing Accountability Patterns using Structural Causal Models
• Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?
• Provenance-Based Assessment of Plans in Context