MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation

Abstract

Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.

References in corpus (9)

• The Future of Digital Health with Federated Learning
• Confounding variables can degrade generalization performance of radiological deep learning models
• Federated Learning Enables Big Data for Rare Cancer Boundary Detection
• Why rankings of biomedical image analysis competitions should be interpreted with care
• TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks
• OpenFL: An open-source framework for Federated Learning
• Common Limitations of Image Processing Metrics: A Picture Story
• GaNDLF: A Generally Nuanced Deep Learning Framework for Scalable End-to-End Clinical Workflows in Medical Imaging
• Industry-Scale Orchestrated Federated Learning for Drug Discovery

Cited by in corpus (8)

• WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis
• The Brain Tumor Segmentation (BraTS) Challenge 2023: Glioma Segmentation in Sub-Saharan Africa Patient Population (BraTS-Africa)
• Federated learning, ethics, and the double black box problem in medical AI
• The Federated Tumor Segmentation (FeTS) Challenge
• Model Ensemble for Brain Tumor Segmentation in Magnetic Resonance Imaging
• Attention-Enhanced Hybrid Feature Aggregation Network for 3D Brain Tumor Segmentation
• When Vocabulary Comprehension Fails Clinical Reasoning: Evaluating Therapy Bots' Safety Risks for Generation Alpha
• GEN-Guard: Correcting Generalization Failures for Deployable Federated Surgical AI