Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Abstract

While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.

References in corpus (15)

• Scikit-learn: Machine Learning in Python
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• SGDR: Stochastic Gradient Descent with Warm Restarts
• CatBoost: gradient boosting with categorical features support
• DARTS: Differentiable Architecture Search
• OpenML: networked science in machine learning
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• ShakeDrop Regularization for Deep Residual Learning
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• NAS evaluation is frustratingly hard
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters

Cited by in corpus (5)

• Best Practices for Scientific Research on Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• LightAutoML: AutoML Solution for a Large Financial Services Ecosystem
• AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data