Distribution-Free Federated Learning with Conformal Predictions

Abstract

Federated learning has attracted considerable interest for collaborative machine learning in healthcare to leverage separate institutional datasets while maintaining patient privacy. However, additional challenges such as poor calibration and lack of interpretability may also hamper widespread deployment of federated models into clinical practice, leading to user distrust or misuse of ML tools in high-stakes clinical decision-making. In this paper, we propose to address these challenges by incorporating an adaptive conformal framework into federated learning to ensure distribution-free prediction sets that provide coverage guarantees. Importantly, these uncertainty estimates can be obtained without requiring any additional modifications to the model. Empirical results on the MedMNIST medical imaging benchmark demonstrate our federated method provides tighter coverage over local conformal predictions on 6 different medical imaging datasets for 2D and 3D multi-class classification tasks. Furthermore, we correlate class entropy with prediction set size to assess task uncertainty.

International Workshop on Trustable, Verifiable and Auditable Federated Learning in Conjunction with AAAI 2022 (FL-AAAI-22)

References in corpus (6)

• On Calibration of Modern Neural Networks
• MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis
• Bayesian Nonparametric Federated Learning of Neural Networks
• Federated Learning for Healthcare Informatics
• Classification with Valid and Adaptive Coverage
• Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction