OpenML Benchmarking Suites

Abstract

Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create and leverage these benchmarking suites. These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R. OpenML benchmarking suites (a) are easy to use through standardized data formats, APIs, and client libraries; (b) come with extensive meta-information on the included datasets; and (c) allow benchmarks to be shared and reused in future studies. We then present a first, carefully curated and practical benchmarking suite for classification: the OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18). Finally, we discuss use cases and applications which demonstrate the usefulness of OpenML benchmarking suites and the OpenML-CC18 in particular.

arXiv commentAccepted for publication in the Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS 2021)

References in corpus (4)

• Scikit-learn: Machine Learning in Python
• SMOTE: Synthetic Minority Over-sampling Technique
• OpenML: networked science in machine learning
• OpenML: An R Package to Connect to the Machine Learning Platform OpenML

Cited by in corpus (15)

• Tunability: Importance of Hyperparameters of Machine Learning Algorithms
• An Open Source AutoML Benchmark
• Decoding machine learning benchmarks
• Post-Contextual-Bandit Inference
• When are Deep Networks really better than Decision Forests at small sample sizes, and how?
• Automatic Exploration of Machine Learning Experiments on OpenML
• ReinBo: Machine Learning pipeline search and configuration with Bayesian Optimization embedded Reinforcement Learning
• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning
• Chameleon: Learning Model Initializations Across Tasks With Different Schemas
• Solving Constrained CASH Problems with ADMM
• Minimizing Close-k Aggregate Loss Improves Classification
• Lifting Interpretability-Performance Trade-off via Automated Feature Engineering
• Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers
• Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning
• Test for non-negligible adverse shifts