Slimmable Neural Networks

Abstract

We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

Accepted in ICLR 2019

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Pruning Filters for Efficient ConvNets
• Learning Structured Sparsity in Deep Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• FiLM: Visual Reasoning with a General Conditioning Layer
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

Cited by in corpus (9)

• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• FasterSeg: Searching for Faster Real-time Semantic Segmentation
• Towards Efficient Training for Neural Network Quantization
• Switchable Precision Neural Networks
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• Channel Equilibrium Networks for Learning Deep Representation
• Frosting Weights for Better Continual Training
• High Frequency Residual Learning for Multi-Scale Image Classification
• Dynamic Multi-path Neural Network