Prioritized Subnet Sampling for Resource-Adaptive Supernet Training
arXiv:2109.05432
Abstract
A resource-adaptive supernet adjusts its subnets for inference to fit the dynamically available resources. In this paper, we propose prioritized subnet sampling to train a resource-adaptive supernet, termed PSS-Net. We maintain multiple subnet pools, each of which stores the information of substantial subnets with similar resource consumption. Considering a resource constraint, subnets conditioned on this resource constraint are sampled from a pre-defined subnet structure space and high-quality ones will be inserted into the corresponding subnet pool. Then, the sampling will gradually be prone to sampling subnets from the subnet pools. Moreover, the one with a better performance metric is assigned with higher priority to train our PSS-Net, if sampling is from a subnet pool. At the end of training, our PSS-Net retains the best subnet in each pool to entitle a fast switch of high-quality subnets for inference when the available resources vary. Experiments on ImageNet using MobileNet-V1/V2 and ResNet-50 show that our PSS-Net can well outperform state-of-the-art resource-adaptive supernets. Our project is publicly available at https://github.com/chenbong/PSS-Net.
References in corpus (8)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- DARTS: Differentiable Architecture Search
- Multi-Scale Dense Networks for Resource Efficient Image Classification
- AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
- MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
- Batch-Shaping for Learning Conditional Channel Gated Networks
- CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment