Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

Abstract

While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few epochs during the main NAS and much larger numbers of epochs during a post-processing step is inefficient due to little correlation in the relative rankings for these two training regimes. To combat both of these problems, we propose to use a recent combination of Bayesian optimization and Hyperband for efficient joint neural architecture and hyperparameter search.

arXiv comment11 pages, 3 figures, 3 tables, ICML 2018 AutoML Workshop

References in corpus (8)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Efficient Architecture Search by Network Transformation
• Shake-Shake regularization
• ShakeDrop Regularization for Deep Residual Learning
• Progressive Neural Architecture Search

Cited by in corpus (40)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• Understanding and Robustifying Differentiable Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• A Constructive Prediction of the Generalization Error Across Scales
• Optimized Generic Feature Learning for Few-shot Classification across Domains
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Scaling Laws for Deep Learning
• Sub-Architecture Ensemble Pruning in Neural Architecture Search
• AutoHAS: Efficient Hyperparameter and Architecture Search
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
• Error-feedback stochastic modeling strategy for time series forecasting with convolutional neural networks
• Speedy Performance Estimation for Neural Architecture Search
• Learnable Embedding Space for Efficient Neural Architecture Compression
• Network Architecture Search for Domain Adaptation
• Feature Partitioning for Efficient Multi-Task Architectures
• A Novel Genetic Algorithm with Hierarchical Evaluation Strategy for Hyperparameter Optimisation of Graph Neural Networks
• HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• An Introduction to Neural Architecture Search for Convolutional Networks
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• Joint Search of Data Augmentation Policies and Network Architectures
• AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data
• AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
• Rethinking Performance Estimation in Neural Architecture Search
• DAAS: Differentiable Architecture and Augmentation Policy Search
• Enhancing network forensics with particle swarm and deep learning: The particle deep framework
• A Sensitivity Analysis of Attention-Gated Convolutional Neural Networks for Sentence Classification
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop
• Multi-objective Neural Architecture Search with Almost No Training
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor
• AmoebaContact and GDFold: a new pipeline for rapid prediction of protein structures