SMASH: One-Shot Model Architecture Search through HyperNetworks

Abstract

Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. Our code is available at https://github.com/ajbrock/SMASH

References in corpus (12)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Layer Normalization
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• Deep Networks with Stochastic Depth
• Swapout: Learning an ensemble of deep architectures
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures

Cited by in corpus (106)

• AutoML: A Survey of the State-of-the-Art
• AutoAugment: Learning Augmentation Policies from Data
• SNAS: Stochastic Neural Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Hierarchical Representations for Efficient Architecture Search
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• Neural Architecture Transfer
• DetNAS: Backbone Search for Object Detection
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• FasterSeg: Searching for Faster Real-time Semantic Segmentation
• Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search
• Auto-Keras: An Efficient Neural Architecture Search System
• How to train your MAML
• Stochastic Hyperparameter Optimization through Hypernetworks
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture
• Hypernetwork functional image representation
• ResKD: Residual-Guided Knowledge Distillation
• Evolving Normalization-Activation Layers
• Language Models with Transformers
• Learning the Effect of Registration Hyperparameters with HyperMorph
• MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
• Searching Central Difference Convolutional Networks for Face Anti-Spoofing
• Adversarially Robust Neural Architectures
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• Contrastive Self-supervised Neural Architecture Search
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• Reinforced Evolutionary Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Meta-Consolidation for Continual Learning
• DSNAS: Direct Neural Architecture Search without Parameter Retraining
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• Auto-Meta: Automated Gradient Based Meta Learner Search
• Meta-Learning Symmetries by Reparameterization
• Federated Neural Architecture Search
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Neural Predictor for Neural Architecture Search
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• Memory-efficient Embedding for Recommendations
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
• HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• DASS: Differentiable Architecture Search for Sparse neural networks
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Densely Connected Search Space for More Flexible Neural Architecture Search
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender System
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation
• SGAS: Sequential Greedy Architecture Search
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Feature Partitioning for Efficient Multi-Task Architectures
• Core-set Sampling for Efficient Neural Architecture Search
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• NAS-Navigator: Visual Steering for Explainable One-Shot Deep Neural Network Synthesis
• Neural Architecture Search For Fault Diagnosis
• IRLAS: Inverse Reinforcement Learning for Architecture Search
• PV-NAS: Practical Neural Architecture Search for Video Recognition
• Fast Neural Architecture Construction using EnvelopeNets
• Combiner and HyperCombiner Networks: Rules to Combine Multimodality MR Images for Prostate Cancer Localisation
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• Neural Architecture Search by Estimation of Network Structure Distributions
• POPNASv2: An Efficient Multi-Objective Neural Architecture Search Technique
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Graph-guided Architecture Search for Real-time Semantic Segmentation
• NAS-Count: Counting-by-Density with Neural Architecture Search
• Searching for Stage-wise Neural Graphs In the Limit
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• Learning Composable Energy Surrogates for PDE Order Reduction
• Fine-Grained Stochastic Architecture Search
• Intra-Ensemble in Neural Networks
• Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?
• Neural Architecture Search Over a Graph Search Space
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• Learn Basic Skills and Reuse: Modularized Adaptive Neural Architecture Search (MANAS)
• Binarized Neural Architecture Search
• Full-attention based Neural Architecture Search using Context Auto-regression
• How Does Supernet Help in Neural Architecture Search?
• Parameter Prediction for Unseen Deep Architectures
• HMCNAS: Neural Architecture Search using Hidden Markov Chains and Bayesian Optimization
• Neural Architecture Search with Random Labels
• Memory-Efficient Differentiable Transformer Architecture Search
• ElixirNet: Relation-aware Network Architecture Adaptation for Medical Lesion Detection
• Single-level Optimization For Differential Architecture Search
• An algorithmic framework for the optimization of deep neural networks architectures and hyperparameters
• Disentangled Neural Architecture Search
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• Evolutionary Algorithm Enhanced Neural Architecture Search for Text-Independent Speaker Verification
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• FDNAS: Improving Data Privacy and Model Diversity in AutoML
• Developing Convolutional Neural Networks using a Novel Lamarckian Co-Evolutionary Algorithm
• Max and Coincidence Neurons in Neural Networks
• Fast Unsupervised Deep Outlier Model Selection with Hypernetworks
• Auto Deep Compression by Reinforcement Learning Based Actor-Critic Structure
• Neural Architecture Search with an Efficient Multiobjective Evolutionary Framework
• Surrogate-assisted Particle Swarm Optimisation for Evolving Variable-length Transferable Blocks for Image Classification
• Towards an unanimous international regulatory body for responsible use of Artificial Intelligence [UIRB-AI]