Locally Free Weight Sharing for Network Width Search

Abstract

Searching for network width is an effective way to slim deep neural networks with hardware budgets. With this aim, a one-shot supernet is usually leveraged as a performance evaluator to rank the performance \wrt~different width. Nevertheless, current methods mainly follow a manually fixed weight sharing pattern, which is limited to distinguish the performance gap of different width. In this paper, to better evaluate each width, we propose a locally free weight sharing strategy (CafeNet) accordingly. In CafeNet, weights are more freely shared, and each width is jointly indicated by its base channels and free channels, where free channels are supposed to loCAte FrEely in a local zone to better represent each width. Besides, we propose to further reduce the search space by leveraging our introduced FLOPs-sensitive bins. As a result, our CafeNet can be trained stochastically and get optimized within a min-min strategy. Extensive experiments on ImageNet, CIFAR-10, CelebA and MS COCO dataset have verified our superiority comparing to other state-of-the-art baselines. For example, our method can further boost the benchmark NAS network EfficientNet-B0 by 0.41\% via searching its width more delicately.

Accepted by ICLR 2021 Spotlight

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• DARTS: Differentiable Architecture Search
• Focal Loss for Dense Object Detection
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• Dynamic Channel Pruning: Feature Boosting and Suppression
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Channel Gating Neural Networks
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• More is Less: A More Complicated Network with Less Inference Complexity
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
• DMCP: Differentiable Markov Channel Pruning for Neural Networks
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

Cited by in corpus (7)

• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets
• Manifold Regularized Dynamic Network Pruning
• BCNet: Searching for Network Width with Bilaterally Coupled Network
• Prioritized Architecture Sampling with Monto-Carlo Tree Search
• SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
• ViTAS: Vision Transformer Architecture Search