Slimmable Neural Networks

Abstract

We present a simple and general method to train a single neural network executable at different widths (number of channels in a layer), permitting instant and adaptive accuracy-efficiency trade-offs at runtime. Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization. At runtime, the network can adjust its width on the fly according to on-device benchmarks and resource constraints, rather than downloading and offloading different models. Our trained networks, named slimmable neural networks, achieve similar (and in many cases better) ImageNet classification accuracy than individually trained models of MobileNet v1, MobileNet v2, ShuffleNet and ResNet-50 at different widths respectively. We also demonstrate better performance of slimmable models compared with individual ones across a wide range of applications including COCO bounding-box object detection, instance segmentation and person keypoint detection without tuning hyper-parameters. Lastly we visualize and discuss the learned features of slimmable networks. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

Accepted in ICLR 2019

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• FitNets: Hints for Thin Deep Nets
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Pruning Filters for Efficient ConvNets
• Learning Structured Sparsity in Deep Neural Networks
• Feature Pyramid Networks for Object Detection
• Revisiting Batch Normalization For Practical Domain Adaptation
• Xception: Deep Learning with Depthwise Separable Convolutions
• Channel Pruning for Accelerating Very Deep Neural Networks
• FiLM: Visual Reasoning with a General Conditioning Layer
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Demystifying Neural Style Transfer
• SkipNet: Learning Dynamic Routing in Convolutional Networks
• BlockDrop: Dynamic Inference Paths in Residual Networks
• Efficient Video Object Segmentation via Network Modulation
• Learning Anytime Predictions in Neural Networks via Adaptive Loss Balancing
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

Cited by in corpus (51)

• Bringing AI To Edge: From Deep Learning's Perspective
• DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• FasterSeg: Searching for Faster Real-time Semantic Segmentation
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
• Human Activity Recognition on Microcontrollers with Quantized and Adaptive Deep Neural Networks
• Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices
• Budget-Aware Adapters for Multi-Domain Learning
• Towards Efficient Training for Neural Network Quantization
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Dynamic Decision Tree Ensembles for Energy-Efficient Inference on IoT Edge Nodes
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Joint Multi-Dimension Pruning via Numerical Gradient Update
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• Weight-dependent Gates for Network Pruning
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Switchable Precision Neural Networks
• Self-Damaging Contrastive Learning
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
• FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training
• Activate or Not: Learning Customized Activation
• Modularity in Deep Learning: A Survey
• Any-Precision Deep Neural Networks
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Structured Convolutions for Efficient Neural Network Design
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• ROMA: Run-Time Object Detection To Maximize Real-Time Accuracy
• Evolving Search Space for Neural Architecture Search
• Binarizing MobileNet via Evolution-based Searching
• Towards Backward-Compatible Representation Learning
• Channel Equilibrium Networks for Learning Deep Representation
• Pufferfish: Communication-efficient Models At No Extra Cost
• NaviSlim: Adaptive Context-Aware Navigation and Sensing via Dynamic Slimmable Networks
• Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systems
• AlphaNet: Improved Training of Supernets with Alpha-Divergence
• Intra-Ensemble in Neural Networks
• Mix Dimension in Poincaré Geometry for 3D Skeleton-based Action Recognition
• Frosting Weights for Better Continual Training
• WeightNet: Revisiting the Design Space of Weight Networks
• Deep Learning Techniques for Compressive Sensing-Based Reconstruction and Inference -- A Ubiquitous Systems Perspective
• High Frequency Residual Learning for Multi-Scale Image Classification
• Shift-and-Balance Attention
• Scale Calibrated Training: Improving Generalization of Deep Networks via Scale-Specific Normalization
• Dynamic Multi-path Neural Network
• Effective Model Compression via Stage-wise Pruning
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better