Transfer Learning with Neural AutoML

Abstract

We reduce the computational cost of Neural AutoML with transfer learning. AutoML relieves human effort by automating the design of ML algorithms. Neural AutoML has become popular for the design of deep learning architectures, however, this method has a high computation cost. To address this we propose Transfer Neural AutoML that uses knowledge from prior tasks to speed up network design. We extend RL-based architecture search methods to support parallel training on multiple tasks and then transfer the search strategy to new tasks. On language and image classification tasks, Transfer Neural AutoML reduces convergence time over single-task training by over an order of magnitude on many tasks.

Cited by in corpus (17)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• A Survey on Neural Architecture Search
• Neural Architecture Transfer
• Design Space for Graph Neural Networks
• Parameter-Efficient Transfer Learning for NLP
• Learning to Branch for Multi-Task Learning
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Fast Task-Aware Architecture Inference
• InstaNAS: Instance-aware Neural Architecture Search
• TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
• HyperSTAR: Task-Aware Hyperparameters for Deep Networks
• Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
• On the Bounds of Function Approximations
• Intelligent Trainer for Model-Based Reinforcement Learning
• Width Transfer: On the (In)variance of Width Optimization
• AutoSmart: An Efficient and Automatic Machine Learning framework for Temporal Relational Data