Progressive Neural Architecture Search

Abstract

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

To appear in ECCV 2018 as oral. The code and checkpoint for PNASNet-5 trained on ImageNet (both Mobile and Large) can now be downloaded from https://github.com/tensorflow/models/tree/master/research/slim#Pretrained. Also see https://github.com/chenxi116/PNASNet.TF for refactored and simplified TensorFlow code; see https://github.com/chenxi116/PNASNet.pytorch for exact conversion to PyTorch

References in corpus (5)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Efficient Architecture Search by Network Transformation
• Dual Path Networks

Cited by in corpus (75)

• AutoAugment: Learning Augmentation Policies from Data
• SNAS: Stochastic Neural Architecture Search
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• TSM: Temporal Shift Module for Efficient Video Understanding
• Path-Level Network Transformation for Efficient Architecture Search
• Deep Learning in Mobile and Wireless Networking: A Survey
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Network Decoupling: From Regular to Depthwise Separable Convolutions
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Semi-Supervised Neural Architecture Search
• Language Models with Transformers
• Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• AGAN: Towards Automated Design of Generative Adversarial Networks
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• Adversarially Robust Neural Architectures
• Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Understanding Neural Architecture Search Techniques
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• Auto-Meta: Automated Gradient Based Meta Learner Search
• Evolving Rewards to Automate Reinforcement Learning
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Fine-Grained Neural Architecture Search
• Learnable Embedding Space for Efficient Neural Architecture Compression
• V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation
• DVOLVER: Efficient Pareto-Optimal Neural Network Architecture Search
• Feature Partitioning for Efficient Multi-Task Architectures
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• Efficient Deep Neural Networks
• DeepWaste: Applying Deep Learning to Waste Classification for a Sustainable Planet
• Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
• Efficient Forward Architecture Search
• Interaction-aware Factorization Machines for Recommender Systems
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• Genetic Deep Learning for Lung Cancer Screening
• Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
• NeuNetS: An Automated Synthesis Engine for Neural Network Design
• Searching for Fast Model Families on Datacenter Accelerators
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• Generalizable Resource Allocation in Stream Processing via Deep Reinforcement Learning
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• Neural Architecture Search Over a Graph Search Space
• Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
• WeNet: Weighted Networks for Recurrent Network Architecture Search
• StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
• RePr: Improved Training of Convolutional Filters
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• NASIB: Neural Architecture Search withIn Budget
• TEA-DNN: the Quest for Time-Energy-Accuracy Co-optimized Deep Neural Networks
• Single-level Optimization For Differential Architecture Search
• DarwinML: A Graph-based Evolutionary Algorithm for Automated Machine Learning
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Composite Binary Decomposition Networks
• MLFriend: Interactive Prediction Task Recommendation for Event-Driven Time-Series Data
• EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
• Layout Design for Intelligent Warehouse by Evolution with Fitness Approximation
• Neural Architecture Search in Embedding Space
• Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition
• Neighbourhood Distillation: On the benefits of non end-to-end distillation
• Cascade Bagging for Accuracy Prediction with Few Training Samples
• Multilayer Dense Connections for Hierarchical Concept Classification
• "You might also like this model": Data Driven Approach for Recommending Deep Learning Models for Unknown Image Datasets