Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search
arXiv:2208.00791 · doi:10.1109/TII.2022.3184700
Abstract
Differentiable neural architecture search (DARTS), as a gradient-guided search method, greatly reduces the cost of computation and speeds up the search. In DARTS, the architecture parameters are introduced to the candidate operations, but the parameters of some weight-equipped operations may not be trained well in the initial stage, which causes unfair competition between candidate operations. The weight-free operations appear in large numbers which results in the phenomenon of performance crash. Besides, a lot of memory will be occupied during training supernet which causes the memory utilization to be low. In this paper, a partial channel connection based on channel attention for differentiable neural architecture search (ADARTS) is proposed. Some channels with higher weights are selected through the attention mechanism and sent into the operation space while the other channels are directly contacted with the processed channels. Selecting a few channels with higher attention weights can better transmit important feature information into the search space and greatly improve search efficiency and memory utilization. The instability of network structure caused by random selection can also be avoided. The experimental results show that ADARTS achieved 2.46% and 17.06% classification error rates on CIFAR-10 and CIFAR-100, respectively. ADARTS can effectively solve the problem that too many skip connections appear in the search process and obtain network structures with better performance.
arXiv comment10 pages, 10 figures Y. Xue and J. Qin, "Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search," in IEEE Transactions on Industrial Informatics, 2022, doi: 10.1109/TII.2022.3184700
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Architecture Search with Reinforcement Learning
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- One weird trick for parallelizing convolutional neural networks
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
Cited by in corpus (4)
- UncertaintyFuseNet: Robust Uncertainty-aware Hierarchical Feature Fusion Model with Ensemble Monte Carlo Dropout for COVID-19 Detection
- Hand bone age estimation using divide and conquer strategy and lightweight convolutional neural networks
- CAE-Net: Generalized Deepfake Image Detection using Convolution and Attention Mechanisms with Spatial and Frequency Domain Features
- MGAS: Multi-Granularity Architecture Search for Trade-Off Between Model Effectiveness and Efficiency