Mutually-aware Sub-Graphs Differentiable Architecture Search

Abstract

Differentiable architecture search is prevalent in the field of NAS because of its simplicity and efficiency, where two paradigms, multi-path algorithms and single-path methods, are dominated. Multi-path framework (e.g. DARTS) is intuitive but suffers from memory usage and training collapse. Single-path methods (e.g.GDAS and ProxylessNAS) mitigate the memory issue and shrink the gap between searching and evaluation but sacrifice the performance. In this paper, we propose a conceptually simple yet efficient method to bridge these two paradigms, referred as Mutually-aware Sub-Graphs Differentiable Architecture Search (MSG-DAS). The core of our framework is a differentiable Gumbel-TopK sampler that produces multiple mutually exclusive single-path sub-graphs. To alleviate the severer skip-connect issue brought by multiple sub-graphs setting, we propose a Dropblock-Identity module to stabilize the optimization. To make best use of the available models (super-net and sub-graphs), we introduce a memory-efficient super-net guidance distillation to improve training. The proposed framework strikes a balance between flexible memory usage and searching quality. We demonstrate the effectiveness of our methods on ImageNet and CIFAR10, where the searched models show a comparable performance as the most recent approaches.

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Efficient Neural Architecture Search via Parameter Sharing
• DropBlock: A regularization method for convolutional networks
• SNAS: Stochastic Neural Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Hierarchical Representations for Efficient Architecture Search
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
• NAS-FCOS: Fast Neural Architecture Search for Object Detection
• Differentiable Architecture Search with Ensemble Gumbel-Softmax
• Cyclic Differentiable Architecture Search
• Neural Architecture Search with Random Labels