On Leakage in Machine Learning Pipelines

Abstract

Machine learning (ML) provides powerful tools for predictive modeling. ML's popularity stems from the promise of sample-level prediction with applications across a variety of fields from physics and marketing to healthcare. However, if not properly implemented and evaluated, ML pipelines may contain leakage typically resulting in overoptimistic performance estimates and failure to generalize to new data. This can have severe negative financial and societal implications. Our aim is to expand understanding associated with causes leading to leakage when designing, implementing, and evaluating ML pipelines. Illustrated by concrete examples, we provide a comprehensive overview and discussion of various types of leakage that may arise in ML pipelines.

arXiv commentsecond draft

References in corpus (8)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• Model Cards for Model Reporting
• Assessing and tuning brain decoders: cross-validation, caveats, and guidelines
• How I failed machine learning in medical imaging -- shortcomings and recommendations
• Challenges in Deploying Machine Learning: a Survey of Case Studies
• Evaluating time series forecasting models: An empirical study on performance estimation methods
• Cross-validation: what does it estimate and how well does it do it?
• Bias in Machine Learning Software: Why? How? What to do?

Cited by in corpus (1)

• A geometric and deep learning reproducible pipeline for monitoring floating anthropogenic debris in urban rivers using in situ cameras