Practical Block-wise Neural Network Architecture Generation

Abstract

Convolutional neural networks have gained a remarkable success in computer vision. However, most usable network architectures are hand-crafted and usually require expertise and elaborate design. In this paper, we provide a block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy. The optimal network block is constructed by the learning agent which is trained sequentially to choose component layers. We stack the block to construct the whole auto-generated network. To accelerate the generation process, we also propose a distributed asynchronous framework and an early stop strategy. The block-wise generation brings unique advantages: (1) it performs competitive results in comparison to the hand-crafted state-of-the-art networks on image classification, additionally, the best network generated by BlockQNN achieves 3.54% top-1 error rate on CIFAR-10 which beats all existing auto-generate networks. (2) in the meanwhile, it offers tremendous reduction of the search space in designing networks which only spends 3 days with 32 GPUs, and (3) moreover, it has strong generalizability that the network built on CIFAR also performs well on a larger-scale ImageNet dataset.

Accepted to CVPR 2018

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Striving for Simplicity: The All Convolutional Net
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures

Cited by in corpus (14)

• Searching for Activation Functions
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Hierarchical Representations for Efficient Architecture Search
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Path-Level Network Transformation for Efficient Architecture Search
• Peephole: Predicting Network Performance Before Training
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• Differentiable Architecture Search with Ensemble Gumbel-Softmax
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• Fast Neural Architecture Construction using EnvelopeNets
• Learned Indexes for Dynamic Workloads
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure