Neural Network Architecture Optimization through Submodularity and Supermodularity
arXiv:1609.00074
Abstract
Deep learning models' architectures, including depth and width, are key factors influencing models' performance, such as test accuracy and computation time. This paper solves two problems: given computation time budget, choose an architecture to maximize accuracy, and given accuracy requirement, choose an architecture to minimize computation time. We convert this architecture optimization into a subset selection problem. With accuracy's submodularity and computation time's supermodularity, we propose efficient greedy optimization algorithms. The experiments demonstrate our algorithm's ability to find more accurate models or faster models. By analyzing architecture evolution with growing time budget, we discuss relationships among accuracy, time and architecture, and give suggestions on neural network architecture design.
arXiv commentWithdrawn due to incompleteness and some overlaps with existing literatures, I will resubmit adding further results
References in corpus (5)
- Practical Bayesian Optimization of Machine Learning Algorithms
- Scalable Bayesian Optimization Using Deep Neural Networks
- Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints
- Input Warping for Bayesian Optimization of Non-stationary Functions
- Optimizing Recurrent Neural Networks Architectures under Time Constraints
Cited by in corpus (5)
- Learning Efficient Convolutional Networks through Network Slimming
- Toward a Period-Specific Optimized Neural Network for OCR Error Correction of Historical Hebrew Texts
- Autonomous Deep Learning: A Genetic DCNN Designer for Image Classification
- Autonomously and Simultaneously Refining Deep Neural Network Parameters by a Bi-Generative Adversarial Network Aided Genetic Algorithm
- Autonomously and Simultaneously Refining Deep Neural Network Parameters by Generative Adversarial Networks