Reproducibility in Machine Learning for Health

Abstract

Machine learning algorithms designed to characterize, monitor, and intervene on human health (ML4H) are expected to perform safely and reliably when operating at scale, potentially outside strict human supervision. This requirement warrants a stricter attention to issues of reproducibility than other fields of machine learning. In this work, we conduct a systematic evaluation of over 100 recently published ML4H research papers along several dimensions related to reproducibility. We find that the field of ML4H compares poorly to more established machine learning fields, particularly concerning data and code accessibility. Finally, drawing from success in other fields of science, we propose recommendations to data providers, academic publishers, and the ML4H research community in order to promote reproducible research moving forward.

Presented at the ICLR 2019 Reproducibility in Machine Learning Workshop

References in corpus (6)

• Split learning for health: Distributed deep learning without sharing raw patient data
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Are GANs Created Equal? A Large-Scale Study
• Fair Regression for Health Care Spending
• Rethinking clinical prediction: Why machine learning must consider year of care and feature aggregation
• Generalizability of predictive models for intensive care unit patients

Cited by in corpus (5)

• MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III
• Randomness In Neural Network Training: Characterizing The Impact of Tooling
• A Scalable Approach for Facial Action Unit Classifier Training UsingNoisy Data for Pre-Training
• An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain
• Disease Progression Modeling Workbench 360