Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets

Abstract

To obtain excellent deep neural architectures, a series of techniques are carefully designed in EfficientNets. The giant formula for simultaneously enlarging the resolution, depth and width provides us a Rubik's cube for neural networks. So that we can find networks with high efficiency and excellent performance by twisting the three dimensions. This paper aims to explore the twisting rules for obtaining deep neural networks with minimum model sizes and computational costs. Different from the network enlarging, we observe that resolution and depth are more important than width for tiny networks. Therefore, the original method, i.e., the compound scaling in EfficientNet is no longer suitable. To this end, we summarize a tiny formula for downsizing neural architectures through a series of smaller models derived from the EfficientNet-B0 with the FLOPs constraint. Experimental results on the ImageNet benchmark illustrate that our TinyNet performs much better than the smaller version of EfficientNets using the inversed giant formula. For instance, our TinyNet-E achieves a 59.9% Top-1 accuracy with only 24M FLOPs, which is about 1.9% higher than that of the previous best MobileNetV3 with similar computational cost. Code will be available at https://github.com/huawei-noah/ghostnet/tree/master/tinynet_pytorch, and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/tinynet.

arXiv commentNeurIPS 2020

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• DARTS: Differentiable Architecture Search
• Training Binary Neural Networks through Learning with Noisy Supervision
• Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
• Positive-Unlabeled Compression on the Cloud
• Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection
• GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
• A Semi-Supervised Assessor of Neural Architectures

Cited by in corpus (11)

• A Survey on Visual Transformer
• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
• An overview of artificial intelligence techniques for diagnosis of Schizophrenia based on magnetic resonance imaging modalities: Methods, challenges, and future works
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Dynamic Neural Networks: A Survey
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework
• GhostSR: Learning Ghost Features for Efficient Image Super-Resolution
• Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs
• BCNet: Searching for Network Width with Bilaterally Coupled Network
• Out-of-distribution data supervision towards biomedical semantic segmentation