MixPath: A Unified Approach for One-shot Neural Architecture Search
arXiv:2001.05887
Abstract
Blending multiple convolutional kernels is proved advantageous in neural architecture design. However, current two-stage neural architecture search methods are mainly limited to single-path search spaces. How to efficiently search models of multi-path structures remains a difficult problem. In this paper, we are motivated to train a one-shot multi-path supernet to accurately evaluate the candidate architectures. Specifically, we discover that in the studied search spaces, feature vectors summed from multiple paths are nearly multiples of those from a single path. Such disparity perturbs the supernet training and its ranking ability. Therefore, we propose a novel mechanism called Shadow Batch Normalization (SBN) to regularize the disparate feature statistics. Extensive experiments prove that SBNs are capable of stabilizing the optimization and improving ranking performance. We call our unified multi-path one-shot approach as MixPath, which generates a series of models that achieve state-of-the-art results on ImageNet.
ICCV2023
References in corpus (12)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Architecture Search with Reinforcement Learning
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- MixConv: Mixed Depthwise Convolutional Kernels
- Understanding and Robustifying Differentiable Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- GhostNet: More Features from Cheap Operations
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search
- Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
- SGAS: Sequential Greedy Architecture Search
Cited by in corpus (6)
- Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
- Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
- S3NAS: Fast NPU-aware Neural Architecture Search Methodology
- Neural Architecture Search as Sparse Supernet
- Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator
- One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking