NAT: Neural Architecture Transformer for Accurate and Compact Architectures

Abstract

Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-searched architecture may still contain many non-significant or redundant modules or operations (e.g., convolution or pooling), which may not only incur substantial memory consumption and computation cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computation cost. Unfortunately, such a constrained optimization problem is NP-hard. To make the problem feasible, we cast the optimization problem into a Markov decision process (MDP) and seek to learn a Neural Architecture Transformer (NAT) to replace the redundant operations with the more computationally efficient ones (e.g., skip connection or directly removing the connection). Based on MDP, we learn NAT by exploiting reinforcement learning to obtain the optimization policies w.r.t. different architectures. To verify the effectiveness of the proposed strategies, we apply NAT on both hand-crafted architectures and NAS based architectures. Extensive experiments on two benchmark datasets, i.e., CIFAR-10 and ImageNet, demonstrate that the transformed architecture by NAT significantly outperforms both its original form and those architectures optimized by existing methods.

arXiv commentThis paper is accepted by NeurIPS 2019

Cited by in corpus (24)

• A Survey on Visual Transformer
• Discrimination-aware Network Pruning for Deep Model Compression
• Deep Learning for Image Super-resolution: A Survey
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• Dense Regression Network for Video Grounding
• Generative Low-bitwidth Data Free Quantization
• A Thorough Comparison Study on Adversarial Attacks and Defenses for Common Thorax Disease Classification in Chest X-rays
• Contrastive Neural Architecture Search with Neural Architecture Comparators
• Towards Accurate Text-based Image Captioning with Content Diversity Exploration
• Location-aware Graph Convolutional Networks for Video Question Answering
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
• Disturbance-immune Weight Sharing for Neural Architecture Search
• Intelligent Home 3D: Automatic 3D-House Design from Linguistic Descriptions Only
• Online Adaptive Asymmetric Active Learning with Limited Budgets
• Rapid Model Architecture Adaption for Meta-Learning
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
• Towards Accurate and Compact Architectures via Neural Architecture Transformer
• Differentiable Neural Architecture Transformation for Reproducible Architecture Improvement
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• Improving Generative Adversarial Networks with Local Coordinate Coding
• Content-Aware Convolutional Neural Networks