Fast Task-Aware Architecture Inference
arXiv:1902.05781
Abstract
Neural architecture search has been shown to hold great promise towards the automation of deep learning. However in spite of its potential, neural architecture search remains quite costly. To this point, we propose a novel gradient-based framework for efficient architecture search by sharing information across several tasks. We start by training many model architectures on several related (training) tasks. When a new unseen task is presented, the framework performs architecture inference in order to quickly identify a good candidate architecture, before any model is trained on the new task. At the core of our framework lies a deep value network that can predict the performance of input architectures on a task by utilizing task meta-features and the previous model training experiments performed on related tasks. We adopt a continuous parametrization of the model architecture which allows for efficient gradient-based optimization. Given a new task, an effective architecture is quickly identified by maximizing the estimated performance with respect to the model architecture parameters with simple gradient ascent. It is key to point out that our goal is to achieve reasonable performance at the lowest cost. We provide experimental results showing the effectiveness of the framework despite its high computational efficiency.
References in corpus (7)
- Practical Bayesian Optimization of Machine Learning Algorithms
- DARTS: Differentiable Architecture Search
- Large-Scale Evolution of Image Classifiers
- Peephole: Predicting Network Performance Before Training
- Probabilistic Matrix Factorization for Automated Machine Learning
- Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs
- Transfer Learning with Neural AutoML
Cited by in corpus (8)
- A Survey on Neural Architecture Search
- Neural Architecture Transfer
- Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
- AutoML using Metadata Language Embeddings
- FNAS: Uncertainty-Aware Fast Neural Architecture Search
- HyperSTAR: Task-Aware Hyperparameters for Deep Networks
- Privileged Zero-Shot AutoML
- Ranking architectures using meta-learning