AutoGrow: Automatic Layer Growing in Deep Convolutional Networks

Abstract

Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts. Our AutoGrow is efficient. It discovers depth within similar time of training a single DNN. Our code is available at https://github.com/wenwei202/autogrow.

arXiv commentKDD 2020

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Pruning Filters for Efficient ConvNets
• Efficient Neural Architecture Search via Parameter Sharing
• Large-Scale Evolution of Image Classifiers
• Learning Structured Sparsity in Deep Neural Networks
• Efficient Architecture Search by Network Transformation
• Hierarchical Representations for Efficient Architecture Search
• Lifelong Learning with Dynamically Expandable Networks
• Convolutional Neural Fabrics
• Net2Net: Accelerating Learning via Knowledge Transfer
• Path-Level Network Transformation for Efficient Architecture Search
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Modularized Morphing of Neural Networks
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Nonparametric Neural Networks

Cited by in corpus (4)

• Scaling Laws for Neural Language Models
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• Compact CNN Structure Learning by Knowledge Distillation
• Differentiable Network Adaption with Elastic Search Space