Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio

Abstract

Automatic designing computationally efficient neural networks has received much attention in recent years. Existing approaches either utilize network pruning or leverage the network architecture search methods. This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs, so that under each network configuration, one can estimate the FLOPs utilization ratio (FUR) for each layer and use it to determine whether to increase or decrease the number of channels on the layer. Note that FUR, like the gradient of a non-linear function, is accurate only in a small neighborhood of the current network. Hence, we design an iterative mechanism so that the initial network undergoes a number of steps, each of which has a small `adjusting rate' to control the changes to the network. The computational overhead of the entire search process is reasonable, i.e., comparable to that of re-training the final model from scratch. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach, which consistently outperforms the pruning counterpart. The code is available at https://github.com/danczs/NetworkAdjustment.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Wide Residual Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Pruning Filters for Efficient ConvNets
• Efficient Neural Architecture Search via Parameter Sharing
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Learning Structured Sparsity in Deep Neural Networks
• Aggregated Residual Transformations for Deep Neural Networks
• Deep Networks with Stochastic Depth
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• DetNAS: Backbone Search for Object Detection
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Network Pruning via Transformable Architecture Search
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• More is Less: A More Complicated Network with Less Inference Complexity