Optimizing Recurrent Neural Networks Architectures under Time Constraints
arXiv:1608.07892
Abstract
Recurrent neural network (RNN)'s architecture is a key factor influencing its performance. We propose algorithms to optimize hidden sizes under running time constraint. We convert the discrete optimization into a subset selection problem. By novel transformations, the objective function becomes submodular and constraint becomes supermodular. A greedy algorithm with bounds is suggested to solve the transformed problem. And we show how transformations influence the bounds. To speed up optimization, surrogate functions are proposed which balance exploration and exploitation. Experiments show that our algorithms can find more accurate models or faster models than manually tuned state-of-the-art and random search. We also compare popular RNN architectures using our algorithms.
Withdrawn due to incompleteness and some overlaps with existing literatures, I will resubmit adding further results
References in corpus (6)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Practical Bayesian Optimization of Machine Learning Algorithms
- ADADELTA: An Adaptive Learning Rate Method
- Theano: new features and speed improvements
- Gradient-based Hyperparameter Optimization through Reversible Learning
- Architectural Complexity Measures of Recurrent Neural Networks