Layer Flexible Adaptive Computational Time
arXiv:1812.02335
Abstract
Deep recurrent neural networks perform well on sequence data and are the model of choice. However, it is a daunting task to decide the structure of the networks, i.e. the number of layers, especially considering different computational needs of a sequence. We propose a layer flexible recurrent neural network with adaptive computation time, and expand it to a sequence to sequence model. Different from the adaptive computation time model, our model has a dynamic number of transmission states which vary by step and sequence. We evaluate the model on a financial data set and Wikipedia language modeling. Experimental results show the performance improvement of 7\% to 12\% and indicate the model's ability to dynamically change the number of layers along with the computational steps.
arXiv comment11 pages, 5 figures
References in corpus (12)
- Practical Bayesian Optimization of Machine Learning Algorithms
- Neural Architecture Search with Reinforcement Learning
- Training Very Deep Networks
- Designing Neural Network Architectures using Reinforcement Learning
- Adaptive Computation Time for Recurrent Neural Networks
- Convolutional Neural Fabrics
- Conditional Computation in Neural Networks for faster models
- IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
- Self-Delimiting Neural Networks
- Learning to Reason With Adaptive Computation
- Dynamic Prediction Length for Time Series with Sequence to Sequence Networks
- Early Improving Recurrent Elastic Highway Network