OBOE: Collaborative Filtering for AutoML Model Selection

Abstract

Algorithm selection and hyperparameter tuning remain two of the most challenging tasks in machine learning. Automated machine learning (AutoML) seeks to automate these tasks to enable widespread use of machine learning by non-experts. This paper introduces OBOE, a collaborative filtering method for time-constrained model selection and hyperparameter tuning. OBOE forms a matrix of the cross-validated errors of a large number of supervised learning models (algorithms together with hyperparameters) on a large number of datasets, and fits a low rank model to learn the low-dimensional feature vectors for the models and datasets that best predict the cross-validated errors. To find promising models for a new dataset, OBOE runs a set of fast but informative algorithms on the new dataset and uses their cross-validated errors to infer the feature vector for the new dataset. OBOE can find good models under constraints on the number of models fit or the total time budget. To this end, this paper develops a new heuristic for active learning in time-constrained matrix completion based on optimal experiment design. Our experiments demonstrate that OBOE delivers state-of-the-art performance faster than competing approaches on a test bed of supervised learning problems. Moreover, the success of the bilinear model used by OBOE suggests that AutoML may be simpler than was previously understood.

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• Practical Bayesian Optimization of Machine Learning Algorithms
• OpenML: networked science in machine learning
• mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions
• Probabilistic Matrix Factorization for Automated Machine Learning
• A Review of Nonnegative Matrix Factorization Methods for Clustering
• CF4CF: Recommending Collaborative Filtering algorithms using Collaborative Filtering

Cited by in corpus (18)

• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Cost-aware Bayesian Optimization
• Automatic Machine Learning by Pipeline Synthesis using Model-Based Reinforcement Learning and a Grammar
• AutoML using Metadata Language Embeddings
• Matrix Completion with Quantified Uncertainty through Low Rank Gaussian Copula
• SapientML: Synthesizing Machine Learning Pipelines by Learning from Human-Written Solutions
• An ADMM Based Framework for AutoML Pipeline Configuration
• Transfer Learning based Search Space Design for Hyperparameter Tuning
• LightAutoML: AutoML Solution for a Large Financial Services Ecosystem
• Efficient AutoML Pipeline Search with Matrix and Tensor Factorization
• Automatic model training under restrictive time constraints
• Imputation and low-rank estimation with Missing Not At Random data
• Towards A Domain-Customized Automated Machine Learning Framework For Networks and Systems
• How Low Can We Go: Trading Memory for Error in Low-Precision Training
• (Machine) Learning to Improve the Empirical Performance of Discrete Algorithms
• Privileged Zero-Shot AutoML
• Data-driven Algorithm Selection and Parameter Tuning: Two Case studies in Optimization and Signal Processing