Neural Architecture Transfer

Abstract

Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose Neural Architecture Transfer (NAT) to overcome this limitation. NAT is designed to efficiently generate task-specific custom models that are competitive under multiple conflicting objectives. To realize this goal we learn task-specific supernets from which specialized subnets can be sampled without any additional training. The key to our approach is an integrated online transfer learning and many-objective evolutionary search procedure. A pre-trained supernet is iteratively adapted while simultaneously searching for task-specific subnets. We demonstrate the efficacy of NAT on 11 benchmark image classification tasks ranging from large-scale multi-class to small-scale fine-grained datasets. In all cases, including ImageNet, NATNets improve upon the state-of-the-art under mobile settings ($\le$ 600M Multiply-Adds). Surprisingly, small-scale fine-grained datasets benefit the most from NAT. At the same time, the architecture search and transfer is orders of magnitude more efficient than existing NAS methods. Overall, the experimental evaluation indicates that, across diverse image classification tasks and computational objectives, NAT is an appreciably more effective alternative to conventional transfer learning of fine-tuning weights of an existing network architecture learned on standard datasets. Code is available at https://github.com/human-analysis/neural-architecture-transfer

arXiv commentCode is available at https://github.com/human-analysis/neural-architecture-transfer

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• DARTS: Differentiable Architecture Search
• Fine-Grained Visual Classification of Aircraft
• Large-Scale Evolution of Image Classifiers
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• MixConv: Mixed Depthwise Convolutional Kernels
• Random Search and Reproducibility for Neural Architecture Search
• CINIC-10 is not ImageNet or CIFAR-10
• Evaluating the Search Phase of Neural Architecture Search
• FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• Transfer Learning with Neural AutoML
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• Fast Task-Aware Architecture Inference

Cited by in corpus (22)

• SpinalNet: Deep Neural Network with Gradual Input
• A Survey on Evolutionary Computation for Computer Vision and Image Analysis: Past, Present, and Future Trends
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• EvoPruneDeepTL: An Evolutionary Pruning Model for Transfer Learning based Deep Neural Networks
• Efficient Adaptive Ensembling for Image Classification
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Towards Transmission-Friendly and Robust CNN Models over Cloud and Device
• Controllable Pareto Multi-Task Learning
• Exploring Robust Architectures for Deep Artificial Neural Networks
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• Progressive Multi-task Anti-Noise Learning and Distilling Frameworks for Fine-grained Vehicle Recognition
• Teachers Do More Than Teach: Compressing Image-to-Image Models
• Automated Dominative Subspace Mining for Efficient Neural Architecture Search
• Attentional Feature Fusion
• Evolution under Length Constraints for CNN Architecture design
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
• Memory-Efficient Differentiable Transformer Architecture Search
• FaPN: Feature-aligned Pyramid Network for Dense Image Prediction
• Large Scale Neural Architecture Search with Polyharmonic Splines
• Sequential Random Network for Fine-grained Image Classification
• Evolutionary Neural Cascade Search across Supernetworks
• Reconsidering CO2 emissions from Computer Vision