DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers

Abstract

Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. Here, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing, which adaptively slice a part of network parameters for inputs with diverse difficulty levels, while keeping parameters stored statically and contiguously in hardware to prevent the extra burden of sparse computation. Based on this scheme, we present dynamic slimmable network (DS-Net) and dynamic slice-able network (DS-Net++) by input-dependently adjusting filter numbers of CNNs and multiple dimensions in both CNNs and transformers, respectively. To ensure sub-network generality and routing fairness, we propose a disentangled two-stage optimization scheme with training techniques such as in-place bootstrapping (IB), multi-view consistency (MvCo) and sandwich gate sparsification (SGS) to train supernet and gate separately. Extensive experiments on 4 datasets and 3 different network architectures demonstrate our method consistently outperforms state-of-the-art static and dynamic model compression methods by a large margin (up to 6.6%). Typically, DS-Net++ achieves 2-4x computation reduction and 1.62x real-world acceleration over MobileNet, ResNet-50 and Vision Transformer, with minimal accuracy drops (0.1-0.3%) on ImageNet. Code release: https://github.com/changlin31/DS-Net

arXiv commentExtension of the CVPR 2021 oral paper (https://openaccess.thecvf.com/content/CVPR2021/html/Li_Dynamic_Slimmable_Network_CVPR_2021_paper.html)

References in corpus (36)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Categorical Reparameterization with Gumbel-Softmax
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• Do Deep Nets Really Need to be Deep?
• DARTS: Differentiable Architecture Search
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Rethinking the Value of Network Pruning
• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• Pruning Filters for Efficient ConvNets
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• FSSD: Feature Fusion Single Shot Multibox Detector
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• CondConv: Conditionally Parameterized Convolutions for Efficient Inference
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Dynamic Channel Pruning: Feature Boosting and Suppression
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Adaptive Neural Networks for Efficient Inference
• Channel Gating Neural Networks
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
• Adversarial Examples Improve Image Recognition
• BERT Loses Patience: Fast and Robust Inference with Early Exit
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Dynamic Neural Networks: A Survey
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
• Scaling Vision with Sparse Mixture of Experts
• Anytime Neural Prediction via Slicing Networks Vertically
• Dynamic Slimmable Network