Federated learning, ethics, and the double black box problem in medical AI

Abstract

Federated learning (FL) is a machine learning approach that allows multiple devices or institutions to collaboratively train a model without sharing their local data with a third-party. FL is considered a promising way to address patient privacy concerns in medical artificial intelligence. The ethical risks of medical FL systems themselves, however, have thus far been underexamined. This paper aims to address this gap. We argue that medical FL presents a new variety of opacity -- federation opacity -- that, in turn, generates a distinctive double black box problem in healthcare AI. We highlight several instances in which the anticipated benefits of medical FL may be exaggerated, and conclude by highlighting key challenges that must be overcome to make FL ethically feasible in medicine.

References in corpus (10)

• The Future of Digital Health with Federated Learning
• Federated Learning Enables Big Data for Rare Cancer Boundary Detection
• Vertical Federated Learning: Concepts, Advances and Challenges
• Limits of trust in medical AI
• MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
• Federated Learning for Breast Density Classification: A Real-World Implementation
• Grounds for trust: Essential Epistemic Opacity and Computational Reliabilism
• High hopes for "Deep Medicine"? AI, economics, and the future of care
• Diachronic and synchronic variation in the performance of adaptive machine learning systems: The ethical challenges
• A moving target in AI-assisted decision-making: Dataset shift, model updating, and the problem of update opacity