PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search

Abstract

Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-network and searching for an optimal architecture. In this paper, we present a novel approach, namely, Partially-Connected DARTS, by sampling a small part of super-network to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We alleviate it using edge normalization, which adds a new set of edge-level parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoys both faster speed and higher training stability. Experimental results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 with merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) using 3.8 GPU-days for search. Our code has been made available at: https://github.com/yuhuixu1993/PC-DARTS.

Accepted by ICLR2020

References in corpus (12)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• DARTS: Differentiable Architecture Search
• Efficient Neural Architecture Search via Parameter Sharing
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Probabilistic Neural Architecture Search

Cited by in corpus (113)

• AutoML: A Survey of the State-of-the-Art
• NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofing
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Differentiable Quantum Architecture Search
• Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition
• Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture
• EEEA-Net: An Early Exit Evolutionary Neural Architecture Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• Searching Central Difference Convolutional Networks for Face Anti-Spoofing
• Noisy Differentiable Architecture Search
• MANAS: Multi-Scale and Multi-Level Neural Architecture Search for Low-Dose CT Denoising
• Contrastive Self-supervised Neural Architecture Search
• Learning Deep Morphological Networks with Neural Architecture Search
• Rethinking Architecture Selection in Differentiable NAS
• Sample-Efficient Neural Architecture Search by Learning Action Space
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Latency-Aware Differentiable Neural Architecture Search
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• MixPath: A Unified Approach for One-shot Neural Architecture Search
• SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation
• Stronger NAS with Weaker Predictors
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• SGAS: Sequential Greedy Architecture Search
• Analyzing and Mitigating Interference in Neural Architecture Search
• Core-set Sampling for Efficient Neural Architecture Search
• NASGEM: Neural Architecture Search via Graph Embedding Method
• Deep Multimodal Neural Architecture Search
• Cyclic Differentiable Architecture Search
• Differentiable architecture search with multi-dimensional attention for spiking neural networks
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Few-shot Neural Architecture Search
• Hierarchical Neural Architecture Search via Operator Clustering
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• DrNAS: Dirichlet Neural Architecture Search
• Rethinking Graph Neural Architecture Search from Message-passing
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets
• Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• FNAS: Uncertainty-Aware Fast Neural Architecture Search
• NAS-Count: Counting-by-Density with Neural Architecture Search
• ZARTS: On Zero-order Optimization for Neural Architecture Search
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• Contrastive Neural Architecture Search with Neural Architecture Comparators
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• RARTS: An Efficient First-Order Relaxed Architecture Search Method
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• ArchRepair: Block-Level Architecture-Oriented Repairing for Deep Neural Networks
• DAAS: Differentiable Architecture and Augmentation Policy Search
• Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
• Fisher Task Distance and Its Application in Neural Architecture Search
• How Does Supernet Help in Neural Architecture Search?
• Neighborhood-Aware Neural Architecture Search
• Effective, Efficient and Robust Neural Architecture Search
• Operation-level Progressive Differentiable Architecture Search
• NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization
• Generic Neural Architecture Search via Regression
• AutoAdapt: Automated Segmentation Network Search for Unsupervised Domain Adaptation
• Neural Architecture Search with Random Labels
• NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition
• G-DARTS-A: Groups of Channel Parallel Sampling with Attention
• Rethinking Neural Operations for Diverse Tasks
• Hyperparameter optimization with REINFORCE and Transformers
• Analyzing the Expected Hitting Time of Evolutionary Computation-based Neural Architecture Search Algorithms
• HMCNAS: Neural Architecture Search using Hidden Markov Chains and Bayesian Optimization
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Interleaving Learning, with Application to Neural Architecture Search
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• EfficientPose: Efficient Human Pose Estimation with Neural Architecture Search
• Learning by Self-Explanation, with Application to Neural Architecture Search
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• Disentangled Neural Architecture Search
• Direct Federated Neural Architecture Search
• Towards Cardiac Intervention Assistance: Hardware-aware Neural Architecture Exploration for Real-Time 3D Cardiac Cine MRI Segmentation
• What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning
• Neural Architecture Search as Sparse Supernet
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• BNAS-v2: Memory-efficient and Performance-collapse-prevented Broad Neural Architecture Search
• Understanding the wiring evolution in differentiable neural architecture search
• Revisiting Neural Architecture Search
• Effective Model Compression via Stage-wise Pruning
• Partially-Connected Differentiable Architecture Search for Deepfake and Spoofing Detection
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Observations on K-image Expansion of Image-Mixing Augmentation for Classification
• Improving Differentiable Architecture Search with a Generative Model
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• Accelerating Neural Architecture Search via Proxy Data
• Raw Differentiable Architecture Search for Speech Deepfake and Spoofing Detection
• Multi-headed Neural Ensemble Search
• Edge-featured Graph Neural Architecture Search
• Neural Architecture Search in operational context: a remote sensing case-study
• Self-Supervised Neural Architecture Search for Imbalanced Datasets
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS
• On the Security Risks of AutoML
• Guided Evolution for Neural Architecture Search
• Small-Group Learning, with Application to Neural Architecture Search
• Neural Architecture Search via Combinatorial Multi-Armed Bandit
• Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator