An Open Source AutoML Benchmark
arXiv:1907.00909
Abstract
In recent years, an active field of research has developed around automated machine learning (AutoML). Unfortunately, comparing different AutoML systems is hard and often done incorrectly. We introduce an open, ongoing, and extensible benchmark framework which follows best practices and avoids common mistakes. The framework is open-source, uses public datasets and has a website with up-to-date results. We use the framework to conduct a thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.
Accepted paper at the AutoML Workshop at ICML 2019. Code: https://github.com/openml/automlbenchmark/ Accompanying website: https://openml.github.io/automlbenchmark/
References in corpus (4)
Cited by in corpus (17)
- Deep Neural Networks and Tabular Data: A Survey
- AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
- Adaptation Strategies for Automated Machine Learning on Evolving Data
- GAMA: a General Automated Machine learning Assistant
- Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development
- An Adaptive and Near Parameter-free Evolutionary Computation Approach Towards True Automation in AutoML
- AutoML to Date and Beyond: Challenges and Opportunities
- LightAutoML: AutoML Solution for a Large Financial Services Ecosystem
- AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data
- A Neophyte With AutoML: Evaluating the Promises of Automatic Machine Learning Tools
- Bayesian Optimization over Hybrid Spaces
- Ensemble Squared: A Meta AutoML System
- Automated Machine Learning Techniques for Data Streams
- Automatic Componentwise Boosting: An Interpretable AutoML System
- Naive Automated Machine Learning -- A Late Baseline for AutoML
- Man versus Machine: AutoML and Human Experts' Role in Phishing Detection
- AutoDC: Automated data-centric processing