Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search

Abstract

High sensitivity of neural architecture search (NAS) methods against their input such as step-size (i.e., learning rate) and search space prevents practitioners from applying them out-of-the-box to their own problems, albeit its purpose is to automate a part of tuning process. Aiming at a fast, robust, and widely-applicable NAS, we develop a generic optimization framework for NAS. We turn a coupled optimization of connection weights and neural architecture into a differentiable optimization by means of stochastic relaxation. It accepts arbitrary search space (widely-applicable) and enables to employ a gradient-based simultaneous optimization of weights and architecture (fast). We propose a stochastic natural gradient method with an adaptive step-size mechanism built upon our theoretical investigation (robust). Despite its simplicity and no problem-dependent parameter tuning, our method exhibited near state-of-the-art performances with low computational budgets both on image classification and inpainting tasks.

Accepted to ICML 2019. Code is available at https://github.com/shirakawas/ASNG-NAS

Cited by in corpus (24)

• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Search to aggregate neighborhood for graph neural network
• Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
• Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding
• You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient
• Efficient Neural Architecture Search via Proximal Iterations
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• Searching to Exploit Memorization Effect in Learning from Corrupted Labels
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning
• Few-shot Neural Architecture Search
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures
• Rethinking Performance Estimation in Neural Architecture Search
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Automatic Data Augmentation for 3D Medical Image Segmentation
• Searching to Sparsify Tensor Decomposition for N-ary Relational Data
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Single-level Optimization For Differential Architecture Search
• Dynamic Slimmable Network
• Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures
• Single-DARTS: Towards Stable Architecture Search