Neural Architecture Search by Estimation of Network Structure Distributions

Abstract

The influence of deep learning is continuously expanding across different domains, and its new applications are ubiquitous. The question of neural network design thus increases in importance, as traditional empirical approaches are reaching their limits. Manual design of network architectures from scratch relies heavily on trial and error, while using existing pretrained models can introduce redundancies or vulnerabilities. Automated neural architecture design is able to overcome these problems, but the most successful algorithms operate on significantly constrained design spaces, assuming the target network to consist of identical repeating blocks. While such approach allows for faster search, it does so at the cost of expressivity. We instead propose an alternative probabilistic representation of a whole neural network structure under the assumption of independence between layer types. Our matrix of probabilities is equivalent to the population of models, but allows for discovery of structural irregularities, while being simple to interpret and analyze. We construct an architecture search algorithm, inspired by the estimation of distribution algorithms, to take advantage of this representation. The probability matrix is tuned towards generating high-performance models by repeatedly sampling the architectures and evaluating the corresponding networks, while gradually increasing the model depth. Our algorithm is shown to discover non-regular models which cannot be expressed via blocks, but are competitive both in accuracy and computational cost, while not utilizing complex dataflows or advanced training techniques, as well as remaining conceptually simple and highly extensible.

arXiv comment16 pages, 4 figures, 3 tables

References in corpus (19)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Neural Architecture Search with Reinforcement Learning
• Methods for Interpreting and Understanding Deep Neural Networks
• DARTS: Differentiable Architecture Search
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Evaluating the Search Phase of Neural Architecture Search
• Heterogeneous Multilayer Generalized Operational Perceptron
• Probabilistic Neural Architecture Search
• Global optimality conditions for deep neural networks
• Finding Better Topologies for Deep Convolutional Neural Networks by Evolution
• Cyclic Differentiable Architecture Search
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• InstaNAS: Instance-aware Neural Architecture Search