Quantitative Overfitting Management for Human-in-the-loop ML Application Development with ease.ml/meter

Abstract

Simplifying machine learning (ML) application development, including distributed computation, programming interface, resource management, model selection, etc, has attracted intensive interests recently. These research efforts have significantly improved the efficiency and the degree of automation of developing ML models. In this paper, we take a first step in an orthogonal direction towards automated quality management for human-in-the-loop ML application development. We build ease. ml/meter, a system that can automatically detect and measure the degree of overfitting during the whole lifecycle of ML application development. ease. ml/meter returns overfitting signals with strong probabilistic guarantees, based on which developers can take appropriate actions. In particular, ease. ml/meter provides principled guidelines to simple yet nontrivial questions regarding desired validation and test data sizes, which are among commonest questions raised by developers. The fact that ML application development is typically a continuous procedure further worsens the situation: The validation and test data sets can lose their statistical power quickly due to multiple accesses, especially in the presence of adaptive analysis. ease. ml/meter addresses these challenges by leveraging a collection of novel techniques and optimizations, resulting in practically tractable data sizes without compromising the probabilistic guarantees. We present the design and implementation details of ease. ml/meter, as well as detailed theoretical analysis and empirical evaluation of its effectiveness.

References in corpus (7)

• Snorkel: Rapid Training Data Creation with Weak Supervision
• Data Programming: Creating Large Training Sets, Quickly
• Generative Adversarial Networks recover features in astrophysical images of galaxies beyond the deconvolution limit
• Using transfer learning to detect galaxy mergers
• The Ladder: A Reliable Leaderboard for Machine Learning Competitions
• PSFGAN: a generative adversarial network system for separating quasar point sources and host galaxy light
• Continuous Integration of Machine Learning Models with ease.ml/ci: Towards a Rigorous Yet Practical Treatment

Cited by in corpus (2)

• The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice
• Workload Scheduling on heterogeneous Mobile Edge Cloud in 5G networks to Minimize SLA Violation