DAAS: Differentiable Architecture and Augmentation Policy Search

Abstract

Neural architecture search (NAS) has been an active direction of automatic machine learning (Auto-ML), aiming to explore efficient network structures. The searched architecture is evaluated by training on datasets with fixed data augmentation policies. However, recent works on auto-augmentation show that the suited augmentation policies can vary over different structures. Therefore, this work considers the possible coupling between neural architectures and data augmentation and proposes an effective algorithm jointly searching for them. Specifically, 1) for the NAS task, we adopt a single-path based differentiable method with Gumbel-softmax reparameterization strategy due to its memory efficiency; 2) for the auto-augmentation task, we introduce a novel search method based on policy gradient algorithm, which can significantly reduce the computation complexity. Our approach achieves 97.91% accuracy on CIFAR-10 and 76.6% Top-1 accuracy on ImageNet dataset, showing the outstanding performance of our search algorithm.

References in corpus (14)

• Wide Residual Networks
• DARTS: Differentiable Architecture Search
• Efficient Neural Architecture Search via Parameter Sharing
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• ShakeDrop Regularization for Deep Residual Learning
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• DADA: Differentiable Automatic Data Augmentation
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Joint Search of Data Augmentation Policies and Network Architectures
• Direct Differentiable Augmentation Search