Path-Level Network Transformation for Efficient Architecture Search

Abstract

We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that can only perform layer-level architecture modifications, such as adding (pruning) filters or inserting (removing) a layer, which fails to change the topology of connection paths. Our proposed path-level transformation operations enable the meta-controller to modify the path topology of the given network while keeping the merits of reusing weights, and thus allow efficiently designing effective structures with complex path topologies like Inception models. We further propose a bidirectional tree-structured reinforcement learning meta-controller to explore a simple yet highly expressive tree-structured architecture space that can be viewed as a generalization of multi-branch architectures. We experimented on the image classification datasets with limited computational resources (about 200 GPU-hours), where we observed improved parameter efficiency and better test results (97.70% test accuracy on CIFAR-10 with 14.3M parameters and 74.6% top-1 accuracy on ImageNet in the mobile setting), demonstrating the effectiveness and transferability of our designed architectures.

arXiv commentICML 2018

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Densely Connected Convolutional Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Efficient Neural Architecture Search via Parameter Sharing
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• Shake-Shake regularization
• Progressive Neural Architecture Search
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Practical Block-wise Neural Network Architecture Generation

Cited by in corpus (39)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Understanding and Robustifying Differentiable Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Deep Learning for Generic Object Detection: A Survey
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• BRP-NAS: Prediction-based NAS using GCNs
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
• HAQ: Hardware-Aware Automated Quantization with Mixed Precision
• AGAN: Towards Automated Design of Generative Adversarial Networks
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Sub-Architecture Ensemble Pruning in Neural Architecture Search
• Federated Neural Architecture Search
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Memory-efficient Embedding for Recommendations
• Hardware-Centric AutoML for Mixed-Precision Quantization
• Multinomial Distribution Learning for Effective Neural Architecture Search
• An Introduction to Neural Architecture Search for Convolutional Networks
• Prune and Replace NAS
• On the Privacy Risks of Cell-Based NAS Architectures
• Rethinking the Number of Channels for the Convolutional Neural Network
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS
• Binarized Neural Architecture Search for Efficient Object Recognition
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• EENA: Efficient Evolution of Neural Architecture
• EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
• CP-NAS: Child-Parent Neural Architecture Search for Binary Neural Networks
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor