VINNAS: Variational Inference-based Neural Network Architecture Search
arXiv:2007.06103
Abstract
In recent years, neural architecture search (NAS) has received intensive scientific and industrial interest due to its capability of finding a neural architecture with high accuracy for various artificial intelligence tasks such as image classification or object detection. In particular, gradient-based NAS approaches have become one of the more popular approaches thanks to their computational efficiency during the search. However, these methods often experience a mode collapse, where the quality of the found architectures is poor due to the algorithm resorting to choosing a single operation type for the entire network, or stagnating at a local minima for various datasets or search spaces. To address these defects, we present a differentiable variational inference-based NAS method for searching sparse convolutional neural networks. Our approach finds the optimal neural architecture by dropping out candidate operations in an over-parameterised supergraph using variational dropout with automatic relevance determination prior, which makes the algorithm gradually remove unnecessary operations and connections without risking mode collapse. The evaluation is conducted through searching two types of convolutional cells that shape the neural network for classifying different image datasets. Our method finds diverse network cells, while showing state-of-the-art accuracy with up to almost 2 times fewer non-zero parameters.
arXiv comment10 pages
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Neural Architecture Search with Reinforcement Learning
- Neural Architecture Search: A Survey
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Weight Uncertainty in Neural Networks
- Random Erasing Data Augmentation
- Efficient Neural Architecture Search via Parameter Sharing
- Group Sparse Regularization for Deep Neural Networks
- Variational Dropout and the Local Reparameterization Trick
- Bayesian Compression for Deep Learning
- Variational Dropout Sparsifies Deep Neural Networks
- Understanding and Robustifying Differentiable Architecture Search
- Random Search and Reproducibility for Neural Architecture Search
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- Path-Level Network Transformation for Efficient Architecture Search
- Training Neural Networks with Local Error Signals
- SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
- Noisy Differentiable Architecture Search
- Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
- Deep Pyramidal Residual Networks with Separated Stochastic Depth
- AutoSpeech: Neural Architecture Search for Speaker Recognition