Adaptation Strategies for Automated Machine Learning on Evolving Data

Abstract

Automated Machine Learning (AutoML) systems have been shown to efficiently build good models for new datasets. However, it is often not clear how well they can adapt when the data evolves over time. The main goal of this study is to understand the effect of data stream challenges such as concept drift on the performance of AutoML methods, and which adaptation strategies can be employed to make them more robust. To that end, we propose 6 concept drift adaptation strategies and evaluate their effectiveness on different AutoML approaches. We do this for a variety of AutoML approaches for building machine learning pipelines, including those that leverage Bayesian optimization, genetic programming, and random search with automated stacking. These are evaluated empirically on real-world and synthetic data streams with different types of concept drift. Based on this analysis, we propose ways to develop more sophisticated and robust AutoML techniques.

arXiv comment12 pages, 7 figures (14 counting subfigures), submitted to TPAMI - AutoML Special Issue

References in corpus (9)

• Scikit-learn: Machine Learning in Python
• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• A continual learning survey: Defying forgetting in classification tasks
• OpenML: networked science in machine learning
• Characterizing Concept Drift
• Automated Machine Learning: From Principles to Practices
• An Open Source AutoML Benchmark
• Towards AutoML in the presence of Drift: first results

Cited by in corpus (4)

• Challenges in Deploying Machine Learning: a Survey of Case Studies
• Online AutoML: An adaptive AutoML framework for online learning
• Padé Approximant Neural Networks for Enhanced Electric Motor Fault Diagnosis Using Vibration and Acoustic Data
• Wind turbine condition monitoring based on intra- and inter-farm federated learning