DeepArchitect: Automatically Designing and Training Deep Architectures

Abstract

In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture is done manually by the human expert through a slow trial and error process guided mainly by intuition. In this paper we describe a framework for automatically designing and training deep models. We propose an extensible and modular language that allows the human expert to compactly represent complex search spaces over architectures and their hyperparameters. The resulting search spaces are tree-structured and therefore easy to traverse. Models can be automatically compiled to computational graphs once values for all hyperparameters have been chosen. We can leverage the structure of the search space to introduce different model search algorithms, such as random search, Monte Carlo tree search (MCTS), and sequential model-based optimization (SMBO). We present experiments comparing the different algorithms on CIFAR-10 and show that MCTS and SMBO outperform random search. In addition, these experiments show that our framework can be used effectively for model discovery, as it is possible to describe expressive search spaces and discover competitive models without much effort from the human expert. Code for our framework and experiments has been made publicly available.

12 pages, 10 figures. Code available at https://github.com/negrinho/deep_architect. See http://www.cs.cmu.edu/~negrinho/ for more info. In submission to ICCV 2017

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Practical Bayesian Optimization of Machine Learning Algorithms
• Playing Atari with Deep Reinforcement Learning
• Neural Architecture Search with Reinforcement Learning
• Large-Scale Evolution of Image Classifiers

Cited by in corpus (58)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Random Search and Reproducibility for Neural Architecture Search
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• A Survey on Neural Architecture Search
• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• Progressive Neural Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• Probabilistic Neural Architecture Search
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• Contrastive Self-supervised Neural Architecture Search
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Finding Competitive Network Architectures Within a Day Using UCT
• GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• Auto-Meta: Automated Gradient Based Meta Learner Search
• Joint Neural Architecture Search and Quantization
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation
• Neural Feature Search for RGB-Infrared Person Re-Identification
• Soft-Root-Sign Activation Function
• One-Shot Neural Architecture Search via Compressive Sensing
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Evolutionary-Neural Hybrid Agents for Architecture Search
• AutoPose: Searching Multi-Scale Branch Aggregation for Pose Estimation
• AutoSpeech: Neural Architecture Search for Speaker Recognition
• Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation
• Multi-level CNN for lung nodule classification with Gaussian Process assisted hyperparameter optimization
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• Activation Function Optimization Scheme for Image Classification
• Efficient Forward Architecture Search
• Neural Architecture Search for Deep Image Prior
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Neuroevolution in Deep Learning: The Role of Neutrality
• Fast Neural Architecture Construction using EnvelopeNets
• Graph-guided Architecture Search for Real-time Semantic Segmentation
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• NeuNetS: An Automated Synthesis Engine for Neural Network Design
• VEGA: Towards an End-to-End Configurable AutoML Pipeline
• Neural Architecture Search Over a Graph Search Space
• Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?
• EPNAS: Efficient Progressive Neural Architecture Search
• Learned Indexes for Dynamic Workloads
• DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence
• Automated Search for Configurations of Deep Neural Network Architectures
• Explaining Transition Systems through Program Induction
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition
• Evolutionary Algorithm Enhanced Neural Architecture Search for Text-Independent Speaker Verification
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• EIS -- a family of activation functions combining Exponential, ISRU, and Softplus
• Neural Architecture Search via Combinatorial Multi-Armed Bandit