ImmuNeCS: Neural Committee Search by an Artificial Immune System
arXiv:1911.07729
Abstract
Current Neural Architecture Search techniques can suffer from a few shortcomings, including high computational cost, excessive bias from the search space, conceptual complexity or uncertain empirical benefits over random search. In this paper, we present ImmuNeCS, an attempt at addressing these issues with a method that offers a simple, flexible, and efficient way of building deep learning models automatically, and we demonstrate its effectiveness in the context of convolutional neural networks. Instead of searching for the 1-best architecture for a given task, we focus on building a population of neural networks that are then ensembled into a neural network committee, an approach we dub 'Neural Committee Search'. To ensure sufficient performance from the committee, our search algorithm is based on an artificial immune system that balances individual performance with population diversity. This allows us to stop the search when accuracy starts to plateau, and to bridge the performance gap through ensembling. In order to justify our method, we first verify that the chosen search space exhibits the locality property. To further improve efficiency, we also combine partial evaluation, weight inheritance, and progressive search. First, experiments are run to verify the validity of these techniques. Then, preliminary experimental results on two popular computer vision benchmarks show that our method consistently outperforms random search and yields promising results within reasonable GPU budgets. An additional experiment also shows that ImmuNeCS's solutions transfer effectively to a more difficult task, where they achieve results comparable to a direct search on the new task. We believe these findings can open the way for new, accessible alternatives to traditional NAS.
arXiv comment16 pages including references, 6 figures, 3 tables, 2 algorithms
References in corpus (17)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- Neural Architecture Search: A Survey
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Large-Scale Evolution of Image Classifiers
- Efficient Architecture Search by Network Transformation
- Random Search and Reproducibility for Neural Architecture Search
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- Convolutional Neural Fabrics
- Progressive Neural Architecture Search
- Evaluating the Search Phase of Neural Architecture Search
- Best Practices for Scientific Research on Neural Architecture Search
- Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
- A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
- DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence